I think WillyNilly and bah12 have a good point. I don't think anyone is obligated to have someone who has been rude and disrespectful to them at their event.
However, except in a few extreme cases, the "good" half of the couple really isn't that good, because they are letting the "bad" half be rude and disrespectful to their family and friends. They are choosing to stay with this person and if they're doing anything at all to change their behavior behind the scenes, it doesn't seem to be working. Obviously the relationship can be complex and no one outside it can understand all the nuances; but the practical result is, if you link yourself to a rude, boorish person, people may choose to not invite you as a couple, rather than risk you bringing the boor along.
I remember a few threads along these lines--I love my brother, but I can't stand his wife, what can I do? And really, I think all you can do according to etiquette, is put up with both, or put up with neither--especially in the case of a family wedding. I think it is totally within etiquette for the OP's sister to NOT invite the DIL to her wedding. But, this also means she should not have invited her son.
But even if she'd done it "correctly"--not inviting either--there was probably going to be fallout. And I do think that's really sad, especially when there's a grandchild involved who is probably too young to even be aware of what's going on. It's really a decision that each person would have to make for themselves, weighing the consequences of either letting the troublemaker in, or risk not seeing relatives in the future. But I think the social unit does have to be invited, or not, together--not as some obscure etiquette rule, but because in a case like this, they both really do bear some responsibility for how the other one has acted.