I do think the rules about having to invite the whole class, or all the girls, are silly - not all the parents want to host a party of 20+ kids, or are able to. And, in a large class, the sheer number of parties to attend can get silly - a class of 24 kids will be a birthday party every two weeks, on average.
That said, in general, inviting almost everyone from a well defined group to a party is not a good idea, for kids or adults. If you are inviting 50% or less from a group to a party, you're generally okay. On the other extreme, inviting all but one person from a group is very hurtful - it says "I like everyone in this group well enough to invite to my party. Except you. Nyah nyah." That works for an pre-school class, a Guide troupe, or if you're inviting all but one of your coworkers to your wedding.
For kids it's more extreme, because, as the OP said, explaining to a four year old that you can't talk about the party in front of people who weren't invited is a very difficult concept to get across. So the few kids who aren't invited are going to have their noses rubbed in it.
There are some exceptions. I see no need to invite someone who is bullying you, or has been aggressively nasty to you in the past, no matter what the age, for example.