I've been away for a bit so just now catching up on the replies.
I am organizing the lunch kinda by default since Gandalf contacted me and I offered to set it up so he can catch up with a few other people he knows and is working with or has collaborated with. The lunch part is definitely not hosted by the company or being organized by management, it's on our own time. Ferret's separate meeting will be on company time, and the tours of our facilities we're setting up for Gandalf are on company time, but those are targeted, work-specific meetings, not open to all coworkers in the group.
I agree that if this were something where we had invited Gandalf (or he had offered) to come talk to professionals in our field, then I would not even consider excluding Mule as that would be very unprofessional. Or even if it was a general "meet-and-greet one of the big names in our field," Mule would be included. Those two cases would have big invite lists, however, where it would be more fitting for not only Mule but other coworkers who have the same job description, regardless of work group.
It sounds like you're thinking through the right things and this really does seem more social than anything else, so you're fine. Also, if you don't want to say that it's a strictly social lunch, you could define it as a combination social/catchup with getting advice from Gandalf on your specific project. That way, you don't need to worry about Mule, or anyone else, thinking he should have been invited.
I know that most, well everyone, else doesn't agree with me, but please trust me. What you call this is very important.
At a company I used to work for a coworker set up a luncheon with a former CEO, who had retired, that was coming back into town. He invited everyone that worked at the company when the guy was there...that excluded two people. The problem? He called it a mentoring session and one of the two excluded people made a formal complaint to HR about it. In the end, although he was allowed to keep his job, it was after months of investigations, leave without pay, etc. He didn't get reimbursed for his time off, even though he contested it...why? Because he organized the lunch during his working/charging hours and used company email to invite everyone. Also, they used seemingly minor things to prove discrimination that otherwise wouldn't mean anything...sort of how your group doesn't like or respect Mule.
HR told us, at the after action review, that had he simply said it was a social gathering with former coworkers of this CEO and had our coworker made the restaurant reservations and emailed (even on company email) everyone during his lunch hour, then everything would have been ok.
I think that your group wouldn't be torn about this or even mentioned Mule unless there was some small iota of this lunch that leans towards him being invited. Since you really don't want him there, then when you "title" the lunch, be sure that it's social or specific to your project. Any general mentoring or networking opportunities you get will be a bonus...just don't advertise it as such.