After 14 pages, my conclusion is, "It depends."
I have a pretty good handle on who in my circle is vegetarian/vegan/lactose intolerant/gluten intolerant/keeps kosher, etc. As a guest, if I saw a small plate of veggie burgers or gluten-free cookies on an otherwise well-stocked buffet table, I'd instinctively pass them by because I knew they were for someone who needed them.
But that might not have been the case in the party OP describes. Sometimes at large parties a vegetarian option is offered without specific people in mind -- it's more of a "just in case anyone prefers not to eat meat" situation. These "anyones" can be committed vegetarians, people who are thinking about going vegetarian and see an opportunity to sample veggie burgers, people who have just been told by the doctor to cut back on meat, or people who simply don't feel like eating meat that day. Unless the guests all know each other very well, or there is a "reserved for ___" sign on the ___ food, they can't be expected to know that a certain food is for certain people only.
Ideally, every guest should be able to eat his or her fill of whatever s/he likes/is able to eat. But that isn't always the case in real life. Guests with food issues or preferences need to take some responsibility and make some choices. If you're vegetarian and invited to a pig roast, you can decline the invitation, eat beforehand and come for dessert, eat afterward, fill up on sides, or offer to bring something. I'm on Atkins and have left a few potlucks with a less-than-full stomach because so many of the contributions were starch-based. It happens.
All of the above presumes we're talking about a casual party/cookout/buffet/potluck, not a sit-down dinner party.