I've always done best avoiding eating too much fibre. I didn't used to have particular "bad" foods - it was just the overall fibre content of my diet that could make things better or worse. Now I have an ileostomy, and limiting fibre is even more important. Since my stoma blocks so easily, I avoid entirely all the foods known to be the most likely to cause a blockage.
Keeping a food diary is harder than it sounds, as reactions to food may be delayed, and you may react to the overall content of your diet rather than to specific foods. Also the act of eating in general may provoke symptoms, so you may get symptoms after eating one food, but it wouldn't necessarily mean that if you'd eaten anything at that point you wouldn't have got the same symptoms. Finally, remember that symptoms often occur without any factors within our control influencing them, so it's very hard to know if particular foods are giving you problems.
I would look at your overall diet, long-term, and only decide it's worth avoiding a food if it consistently causes reactions. You could also be tested for lactose intolerance to see if that's a problem. You could eliminate foods which are known to be the most common offenders for a few weeks and see if you notice any significant improvements, but the difficulty with this is that nearly every food is labelled as one of the worst offenders by someone, so if you give up all the foods you're told are bad, you'll probably end up giving up practically everything needlessly.