Also, it is generally considered rude to srcutinize or take to close attention to other's eating habits. Even your "only eat it once the vegetarians have made a plate" is requiring guests to rudely track other guest's food consumption. As a hostess, I have the responsiblilty to discreetly make sure all of my guests are well fed. But it's rude of another guest to pay that type of attention.
I see that as "only eat it once the vegetarians have had a CHANCE to make a plate."
If I go through a buffet line, and there's a plate of burgers w/ a little sign that says "vegetarian," I think that as a meat-eater, I should not take one of those burgers on my first time through the line. Ditto anything that's labeled "vegan" or "gluten free." If that's not my restriction, I wouldn't take it. Oh, if the macaroni & cheese were labeled "vegetarian" or the potato salad was "vegan," I'd take an ordinary helping of those, because those are mainstream dishes. But the **alternate** food--reg. burgers & veggie; reg. bread & gluten free; reg. pasta & vegan--I'd wait until the 2nd trip. (I don't think the host should have to make a big announcement if they've labeled the food.) (oh, and if it looked like there was a big pile of the veggie burgers, I might be willing to break this rule of mine.)
And with any hosted buffet (as opposed to a restaurant), *everyone* should wait to make a 2nd trip (or take a 2nd serving, or a serving big enough to be 2) until after enough time has passed that everyone could have gone through once.
Should some other guest get caught up in a conversation and not make it through the line until later, then they're on their own--I agree that those w/ restricted diets have their own half of the equation to hold up.