I used to find that bananas really irritated my Crohns...I guess it's another of case of everyone being different.
Have you tried eating them when they are super spotty ripe? When I used to eat unripen bananas it caused me issues but ripe bananas no issues at all. One day I ate 20 ripe bananas as a test and felt great but if I ate 2 or 3 unripe ones I would have issues.