I was always unsure re which doctor as well. I finally just asked the question to my son's GI. He said I was to contact his office with any concerns, if he felt it was IBD or med related and needed a referral, he would make the referral. If he felt it should be addressed by our GP, he would say so.
I use his advice as a 'general guideline' - with S's issues with a rash and chest pain over the past year, we went to GP first (didn't seem directly related to IBD/med), when GP couldn't offer a solution after a number of visits (ie vague heat rash and vague sore muscles 'diagnosis') and issues did not go away on their own after weeks, I emailed GI and asked if it could be related to IBD/med. In both cases, GI referred to derm and rheumatologist.
I also keep the GI aware of any medical issues that might affect lab results - when S dislocated his shoulder out of the country, I sent GI's office an email as I thought it might affect his CRP (GI's office called me to request a copy of doctor's notes and meds given to S). Also, if we ever need to go to ER for a non-IBD related issue, we always go to GI's hospital and request the info be sent to GI.
I'd suggest you ask your GI's opinion at your next apptmt - it really helped clear up my question re which doctor.