OP, it sounds like you've done the "party hall with split cost" thing only twice now, so it probably won't seem that odd or notable to people if next year you do a potluck, especially if potlucks are the norm for other hosts in your group. (Please correct me if I'm wrong on that.) You could just announce that you're doing a potluck for your holiday party this year, with no reference to the format being different from previous years.
I do think that if people understood there was a $10 charge to attend the party, and they attended the party but didn't pay, that's rude. Whether they thought that charge was unusual, unreasonable, or perfectly fine, they chose to attend, and then chose not to pay.
I can see how you might be disappointed to feel you "have" to change formats because of a couple rude people, but unless you actually say, "It's because some people wouldn't pay up!" I don't think anyone would automatically guess that was your reasoning. At least, I should say that I wouldn't automatically guess that, or in any way speculate about why the party format was changing. If anyone does ask, you could just say you felt like doing something different, or that you're looking forward to your friends' signature dishes, or something else that doesn't involve money at all.
I think the other question was about whether to invite John and Jane to the 2013 party, right? If you're going to make it a potluck, you might look at their track record with other potlucks. If they never bring a dish to any potluck, or contribute in any other way, it might be best for your peace of mind to just leave them off the guest list. If they should ask, you could say something about how it's "just a small party this year" (I think a PP suggested that). You might also want to mention to some of the guests you're closest to, "Hey, I'm not necessarily inviting the whole gang this year because I wanted to downsize the party a bit. So, please be careful who you mention it to."
Alternatively, you could do a "public" (open to all invitees) sign-up sheet for the potluck, with many options for bringing food, dishes, cash, etc., so that everyone can see who's planning to attend without contributing anything. If this plus some gentle nudging ("Jane, I haven't heard what you're bringing yet. Can you tell me so I can put it on the list?") doesn't clue people in, it would probably be best to not invite them in the future.
ETA: Okay, now I'm confused. Is the plan next year for a straight potluck with no cash from guests, or an expectation that people will bring a dish and cash?