Connor's giftedness and weird behavior are entertaining, but they aren't really important tot he etiquette question.
You could use what someone once said to me, when I repeatedly tried to arrange a get-together for our girls (they'd been friends in daycare, and DD really, really loved this girl).
"We're pretty busy with our own family and friends."
The message is, "you aren't one of those people; you aren't our friends."
It wasn't something you can argue with. And it's not really rude. It stung a little bit, but I couldn't really argue with it. And it made me stop trying, because it was clear we'd been defined into a different category now that daycare was ended.
Of course, Jessie will continue to have contact w/ Carol; she should then just continue to be pleasant, etc., and act as though this was no big deal. Bcs it isn't.
There's the less definitive, "Our schedule is really pretty busy; Josh doesn't have time."
I don't think it's fair to put it on Josh and say, "I don't push him to go where he isn't interested"--essentially Jessie would be forcing her son to say, "I don't like Connor." Which would be a rude thing for Josh to say--and he didn't even get to decide to not say it.
There's the "I don't think the boys are that compatible," I suppose. But that's still a bit of a value judgment.