Why not simply state the truth in no uncertain terms?
"We are so glad you live close enough to visit, but I have to say, in all honesty, I really don't care for drop in guests. Just give me a buzz if you want to visit and if I'm home and available we will set it up!"
My DH only had to say this to his extended family once when we moved to their city. No problems - ever. But he changed it to "Jo doesn't care for" - instead of the "I" statements I have there for the OP.
I should also note that I live next door to MIL, and she has a key, and has never used it for anything other than letting the dog out. We also call each other before we go to each other's houses. (We built our homes next door to each other. When MIL found a lot she liked she called me and said, "Jo? We found a lot we like...but it's right next door to you. Is that OK?" I replied with, "Mind your own business and call before you come over. I'll do the same and we will get along just fine.". 14 years later - still working just fine.)
