If you do decide to have another party, I would send out actual invitations and write something like,
"I regret that I will not be able to accommodate additional guests beyond those I've directly invited."
or something like that.
I wouldn't write any reason or excuse such as space or previous year's party not working out because that opens it up to discussion (as in someone trying to convince you that they're just bringing one more person, surely that won't be too many! or some such).
And if anyone whipped out a zip lock bag and proceeded to fill it up at a party I was hosting before the party was even over.... and without even asking first! ... well, I'd have no qualms about saying, "Sorry, these are for the guests of the party, not for to-go packages." I have to say, the zip lock maneuver takes some real nerve!