hi CM - try not to worry about this too much - i've been swimming many times with my stoma, and not once have i had a problem. i make sure the appliance is a reasonably newly applied one (say the day before), and that it's empty before i go into the pool... also i don't eat for a couple of hours beforehand.
i wear a costume which is ruched at the front, so nothing of what's underneath is noticeable.
something for you to investigate - i'm sure i've heard that colostomists can use a kind of stoma cap for swimming purposes, thus not needing to have the full-size appliance on. i have an ileostomy, so this option isn't something i can use, or know much about - maybe have a look on colostomy supplier sites, or ask your stoma nurse.