I don't quite think I'd call it rude, but at the very least, your hosts' behavior sounds kinda clueless.
IMO, if you invite someone to your home, you should at least offer them something in the way of food and/or drink -- but that something can be fairly minimal. In general, tea or coffee and cake, cookies, or crackers are fine.
OTOH, it seems both kind and sensible to tell your guests what to expect. I'm on the East Coast, so the Super Bowl generally coincides with the dinner hour, and all the SB parties I've gone to have involved either dinner (usually a casual, potluck type thing) or, at the very least, substantial appetizers (i.e., more than just a few bags of chips and a bowl of guacamole). Were I to host such a party, I'd tell my guests up front what I was planning, whether that was, "I'm making a huge vat of chili with all the fixings," "I'll have a deli tray and some buffalo wings," or "We'll all chip in to order pizza and wings from Gino's." Perhaps if I were on the West Coast, where kickoff happens at 2 pm, I'd be less inclined to expect or serve a full meal -- but even then, I'd probably make sure my guests knew exactly what I was inviting them to and for. (Obviously, it's not proper to seem more concerned with the food than with the company ... but especially around something like the Super Bowl, I'd hate to put my guests in the tough position of choosing between going hungry because they didn't know to eat beforehand *or* leaving to get dinner elsewhere and missing the game.)
So ... if the game coincided with a mealtime in your area, the hosts probably should have either served dinner or make sure you knew not to expect it (e.g., "We'll have some snacks and desserts"). If it didn't, that doesn't seem quite as bad -- but personally, I'd probably still say something like, "We're going to have a nice dessert bar" rather than just "Come hungry.") If these are new neighbors and otherwise seem like OK people, I probably wouldn't write them off just for this -- but I would be a bit clearer next time extended or accepted an invitation that involved them.