If there's a simple solution, it's this. Since you had more than half of the guest list back out, tell Sue that the party has been cancelled. Have Sue tell Joan. Then, tell Mary, Sue and the other couple that the original party was cancelled, and that frees up your schedule for a party held coincidentally at the same time and place. Make sure Sue doesn't pass the new invitation on to Joan. Due to Joan's absence, I sense that Mary and husband, and the other couple, might find a gap in their schedule.
Sure, it's functionally the same as uninviting Joan, but honestly, it would be more rude to advise her why the guest backed out, or to have her to your house with just you two, Sue and husband, and her and risk your tempers getting out of hand. Considering you have no contact with her socially and no desire to, I'd consider that a less risky proposition.
Virg