I don't think I "judge" anybody on dietary choices -- and I appreciate Drunken Housewife's struggle (filed under Why I Am Not A Raw Vegan; I try, I lose) -- but I do object to people who regularly, even if infrequently, eat meat, fish and/or fowl claiming to be vegetarian and screwing up MY LIFE by making MY DIET a topic of conversation when I turn down the fish or chicken or chili con carne or beef stew (no, I can't just "pick it out") because these people have changed the definition of Vegetarian or Vegan to suit their own ego needs, rather than using the actual definitions of the words. I don't think that applies to OP, though I would prefer s/he use "primarily veg" for clarity, and I don't think it applies at all to Drunken Housewife.
DH, it is perfectly OK with me if you eat seafood occasionally and if someone says, "I thought you were veg," to answer "yeah, but every five years or so I can't resist a lobster." Or, "unless I am in the Famous Seafood Region, when I eat seafood sometimes." I absolutely do not have a problem with this -- it is clear and honest. For OP to say, "I am very careful of the fish I eat, and I don't eat fish very often anyway. It is too much of a pain to explain to people, as I eat vegetarian almost exclusively, and that is what I say, as I can eat anything veg." Clear. Honest. It is the misrepresentation that gets to me, and the bad ways it affects my life.
The "I need protein" thing is specious. All that means is the speaker needs to eat protein, which comes complete from plant sources like soybeans and quinoa, and through plant food combinations, like beans and rice. That has nothing to do with needing MEAT, FISH OR FOWL, and shows a lack of knowledge of vegetarian/vegan cooking and nutrition. I see it as a excuse and fundamentally dishonest. So I guess I am "judging" on the question of fundamental honesty.
Nobody, ever, should have to argue about or justify what they eat or don't eat, especially and particularly in a social context. It is rude to comment at table about what others do or don't eat, generally (there are a very few exceptions) and I solidly object to people starting arguments with me over what I eat, waving meat under my nose, justifying their own food choices, and generally making ME uncomfortable and MY experience unpleasant. Especially since I, as a general rule, Do. Not. Care. what others eat, do not wish to discuss it, be attacked about it, listen to their justifications and so on. And I hate it when another veg/vegan pulls out a soapbox just as much, because that is equally rude and unpleasant. I just want everybody to shut up and eat. I am not pushy or preachy at all -- I grew up veg and have never seen a reason to change that -- and just want to eat what I want to eat and extend others the same quiet, unremarkable, courtesy. So I guess I am "judging" on common courtesy, too.
If you're coming to my house for dinner, I do want to know if you have any other limitations, so I can work around them, so I will probably ask. And if I see you doing something else later, such as eating sugar or gluten, I will most likely think you just changed your mind, and will ask again the next time I invite you over. But I will not make an issue of it, ever. That is the rude part, IMO.