I have terrible anxiety. I think my flight or fight response is just plain broken. Anxiety is my response to pretty much everything, appropriate or not. It used to be utterly debilitating, there were a few years when it actually kept me out of school. It's better now, but I also know better how to deal with it.
Caffeine is the worst thing in the world for me. I can't even have anything decaffeinated, because there is still some caffeine in it. A decaf chai had my heart rate accelerated for 8 hours.
The fact that your symptoms are showing up whenever you go to college (i.e. at the same stimulus) points to anxiety to me. If that were happening to me, I would have to ask myself what is it about college that is causing my anxiety. Such an extreme anxious response usually has a reason and can be your body's way of telling you that there is something you are subconsciously unhappy about. I obviously don't know what the cause is for you and I won't presume to psychoanalyze you, but if you suspect you are having panic attacks, it might be worth it to ask yourself why. If it were me, I would ask myself if I was unhappy with the college in general, or if I was having second thoughts about my major, stuff like that.