I feel differently. For one thing, lot of people who are diagnosed with IBS turn out to actually have IBD or some other problem.
The main reason I'm not bothered by the comparisons is that I think people with IBS have a hard time because it's not a diagnosis that is taken seriously. Doctors seem to use it as a way of saying there's nothing really wrong with a patient. They imply that people cause it themselves through lifestyle choices. And since it's supposed to be related to stress, it can imply something negative about a person's character. It's not a valid disease, and so people diagnosed with it aren't given the unspoken permission to have problems managing it, which they are probably more likely to get with IBD.
I've seen quite a few posts on this forum from people who were previously diagnosed with IBS (as I was) or currently diagnosed with it but are here because they suspect they may have Crohn's who have experienced this kind of thinking, especially from medical professionals. And when it comes to the embarrassment and social factors, IBS could probably match some cases of IBD.
If it comes down to general levels of health, absolutely people with IBS have it easier. Severity of symptoms will generally be much worse for people with IBD, though there are probably exceptions (mild Crohn's or severe IBS). Operations, medications, etc. are all negatives that people with IBS don't have to deal with. But I wouldn't be too bothered by the comparisons, even if they are inaccurate, because I think many people with IBS have a hard time in other ways. Or at least I wouldn't be any more annoyed by someone with IBS thinking they're as sick as someone with IBD than by other situations in which inaccurate assumptions are made about IBD.