I don't know that death is the only factor, flu hits a healthy person pretty hard let alone someone immuno-compromised.
If you can afford to take time off work, and have someone who can make you food, look after your kids etc then you've not got much to worry about (other than the inevitable, painful battle to get your guts back to normal after any kind of bug).
If on the other hand you are single, with a terrible boss then you've got something to worry about!
I always get the shot now because I would rather be protected against three strains of flu than none. And it always hits me pretty darn hard. Last time I had it (I swear it was swine flu before the vaccine came out) I was in bed 2 weeks, no kidding. I won't even go into how long it took before I could eat again, a normal cold usually has my guts throw a fit for 1-2 weeks. That first hot meal which stays in after illness is always real good!
This topic has come up alot of times before, so just run a search and you can see people's experiences.