Here's the thing. Making a sudden exit from a social event does imply censure/rejection of the host as well as the objectionable guest. It does carry the implication that BIL has done something wrong in inviting him.
So, to try to see what's reasonable, OP, I'm trying to look at this from BIL's point of view.
If somebody posted a question like this:
"Our whole family usually gathers on Boxing Day - me and my 3 brothers and our families, my mom, and even my dad and step-mom. I know not every family can do that, but I feel like we're very lucky everybody can get along so well, and it's an important tradition to us.
Here's the problem - Dad and Stepmom have been real you-know-whats to my youngest brother and his bride. I really don't want to take sides or get put in the middle, but I hear from Mom that they have decided to cut him off entirely. I can't say it's unjustified, but I am really hoping we can all spend some time together on neutral ground. What do I do? If younger brother starts asking whether Dad will attend, or -God forbid- walks out of the party. Should I just not invite Dad?"
I think the general consensus on the board would likely be, that as host he should invite whoever he wants to invite, answer any direct questions truthfully, and let the guests make their own decisions about attending or leaving. He would probably get some advice as well, about how it would be kind but not necessary to offer flexible times to meet with you.
I think there is a difference between you asking whether FIL and SM are invited, versus telling your feelings/decision to BIL and expecting him to change his plans accordingly. I think making a grand announcement that "we are estranged!" makes it BIL's problem, which it should not be.
If however, you asked about FIL/SM being invited, and said, "Well, in that case could we come over earlier, or get together with you the next day, because we're not comfortable staying if FIL/SM come", then you are taking responsibility for managing your own problems, and offering BIL a reasonable modification, and showing you're not looking to punish or keep your distance from the rest of the family.
From what I remember of the backstory, even if BIL invites him, there's no guarantee that FIL would even show. And it would be a shame to miss out on the family just because of something that MIGHT happen.