Not rude not to drink, and don't feel like you have to have some sort of beverage in your hand. I have a group of HS friends who all live near each other, but I'm about 45 mins away. we get together every now and then, with me usually driving down to one of their homes. I've stayed over, so I can have more than one drink, or had one glass of wine, then switched to water, and no one bats and eye. we are all mature adults, and no one wants to be in the position of not being able to drive home.
that being said, I do hate when people comment and give you grief. Many moons ago, I was dragged by friends to one of their cousins' Halloween bashes. I knew no one except my friends, but did know the hosts etc. like to party hearty. So we walk in, and I'm exhausted from having moved the prior week, and not at all in a festive mood. When asked what I wanted to drink, i asked for a soda, and was basically heckled. thankfully, i never had to see this group again, and my other friends were not all that impressed with the host, etc ether.