Depending on what specifically your nephew's SO has done to your sister (it'd have to be pretty severe), I think she is within her rights to invite her son only and not his SO.
Your sister, however, has to live with the consequences of making that decision (including losing contact with her son and her grandchild, if that is the consequence her son and SO choose).
Pesonally, I'd stay as far away from this drama as physically possible. If my sister were acting this way, I'd stay so far away that I'd probably decline the invite altogether.
Well, from what I understand: My sister and her husband to be, were invited to the baptism - and were rudely treated by the girlfriend, in favor of her eldest son's former step-mother (both my sister and this former step-mother are now divorced from eldest son's father). Former step-mother and future DIL caused a huge scene at the baptism, embarrassing my sister (the baby's grandmother). No other family was invited to this. (All of them, except my brother and I, live in the immediate area.)
No others in my family, including my sister (the baby's grandmother) was invited to the child's first birthday, have been excluded from holiday celebrations, etc.
On one occasion, when my sister and future husband visited to see eldest son, baby's mother and baby, the baby's mother "hid out" at a neighbor's house, while my sister was visiting, leaving eldest son to lie about her being at work. It was later revealed that baby's mother did not wish to spend any time in the same place as my sister.
Any toys or gifts given to the baby from my sister, have been either dumped back on her doorstep - or she has been told, thrown away.
Toys given from my sister's ex-husband/ex-husband's family (including second ex-wife/stepmother of eldest son) have been accepted with much gushing.
In the early days of their relationship
, and even while this girl was pregnant, my sister was very good to her. She repaid that kindness with the ultimate slap. And is now insisting that eldest son (and their child) have nothing to do with his mother/baby's grandmother.
This girl has even been snippy with me, through posts on Facebook I've made to my nephew, even though we've never met (the girlfriend and I) in person, nor have I ever had any other interaction with her. I did not engage her in any discussion of anything. Nor will I. I also haven't posted anything further on my nephew's page - on any subject.