Hmmm. Here's my take on it, being unmarried, so not an issue for me. My parents moved from NJ to VA about 8 years ago, and immediately joined a number of social groups. Together, and men's only and ladie's only. Many of the co-ed groups met in people's homes, on a rotating basis, and my parent's split the responsibilty of "prepping" the house, food etc.
As for the single-sex social groups, sometimes they met elsewhere, as in they both belonged to a ladies/men's lunch out group, meeting at a different restaurant each time. But each also belonged to a book group, which met in someone's home. At least with my parents, they'd jointly get the house ready, and the other was welcome to be in the home, partake in snacks, say hello and schmooze a bit, but then would retire elsewhere and let the group do their thing. Having met many of the members of all their groups, I don't think any of them would have done what the LW's group did. Sure, maybe some of the spouses left the house during the group meeting, but on their own, not beacuse they were told to get out.
As far as the LW, Ithink it depends on how the rest feel; if they don't like the way things are run, they're free to leave, or if he's the only one, then he can leave.