I feel like Sis' leftover policy is a manifestation of the invitee's intentions. If you can't be there physically, but you're there in spirit, she recognizes that wish with a plate of food, almost like saying 'we celebrated with you.' If you decline, you weren't, so she doesn't.
I think this is tricky, because it's unclear (and not something you can just ask) is if him not liking a crowd is the main reason or a convenient excuse. He turned down the invitation, however, and from the OP I got more "can't be bothered" than "I wish I could, but social phobia."
Maybe it's just my own personal hang-ups, but if someone declined a thanksgiving invitation, I'd assume they made other plans (Even if those plans are a marathon of Breaking Bad and takeaway). No hard feelings, but I'd personally feel like a line cook and taken for granted if I were expected to give them a plate of food I labored over while they couldn't be bothered to even tolerate my presence for a few hours.