He wants to come to the event, but he doesn't want to go by himself since this is his first one and Kay doesn't want to go, so she keeps asking me if he can go with me.
If he does go, I probably will run into him and say hi (it's that small), but there is no way I'm inviting him to go with me or allow him to gate crash my "me" time.
May I encourage you to really work even harder at getting your head around the concept that you do not have to "agree with everyone and everything" and that your are "allowed to say no and not have to justify why". Not only do your DESERVE this, I am afraid you're right and you will run into Mr.Kay at this event. And I'm afraid shaking him off in person will take some confidence. If you plan ahead, you can 'pay it ahead' with some extra confidence so that when faced with an unplanned tag-a-long, you can handle him.
All you have to do is greet him briefly and then smile and say "You'll need to excuse me now..." as you walk off. Don't be afraid to do it every time you run into him, either!
So my suggestion is that you push your head a bit more and remember that YOU are important and YOU deserve to make your own choices. Also, IMHO, some people use other people's niceness, desire to avoid hurting people and manners, to enable the use of that nice, kind and polite person.
So once you've said "NO, I'm sorry that won't be possible..." several times and/or "Excuse me...." to someone who just keeps coming at you and/or if they keep pressing you with "WHY?" I'd say unless there a real good reason to avoid honesty (your boss or family connections) BE HONEST and says, gently and politely "It's not possible because I've decided it's not possible..."
I also do not think answering (politely and gently) that the 'why' or subject matter is personal business I will not discuss, is in the least bit rude. When someone kept pressing me after that, I finally responded to "It's none of your business...". PB