Yes, to me, the interesting shade is, should you allow non-invited people (like other members of your household, who aren't actually "invited" to this meal) to come in and start grazing, even though an invited person hasn't shown up yet? Like, for how long should you fend people off, saying, "No, this is Late Lucy's portion!" Or alternatively, what if it was something that didn't keep well, and you had the brilliant idea of offering it to your neighbor, who was out in their yard at the time? Should you hold back some of it for Late Lucy, or give it all to the neighbor?
In other words, it's not that (from a generous starting amount) the other invited people just happened to eat everything, or all of a few choice things; it's that the host is actively making a choice to get rid of leftovers, when one invited person hasn't shown up yet (but precedent says they probably will, just late). The former would be natural consequences, but the latter is a bit less natural. Or could be. I would say allowing other members of your household to eat the food is on the more natural end of the spectrum; whereas saying, "I dunno, normally I like leftovers but I'm not feeling them today, everyone grab a dish and let's head to the dumpster," seems more forced.
I remember a thread about someone who attended a meeting/potluck at someone's house, and the home owner invited several of their own family members to eat, even though those people were not involved in the activity at all, and some of them didn't even live there. Like spouse, adult daughter and her spouse, etc.. I think if extra people end up eating food intended for Group X, such that some of Group X doesn't get any, that's a problem. But if Group X have had their fill, it's less dodgy to then let the extra people have at it (though a potluck complicates things as perhaps Betty wanted to take her own leftover casserole home, but Carla is glad to get rid of hers). In the OP's case, you have one member of Group X who is significantly behind the others timewise, so how much consideration do they get?