I think if you can tolerate them (ie they don't make your symptoms worse) then you're ok to have them!
Most nuts give me diarrhoea (I can have ground nuts or nut butters, it's the texture that bothers my tummy).
Just to be on the safe side, I would cut them out for a while, then add them back in and see if there is a difference in your symptoms.