I'm somewhere in the middle. If a guest came over and he turned out to be terrified of cats, I'd try to keep the kitty away from him as much as possible, but it probably wouldn't be a long visit, nor would I suggest get-togethers at my house.
On the other hand, I would move my cat if he was bothering a guest. My cat is allowed on the couches and chairs. He doesn't show much interest in the counters. He's not allowed on the table while I'm eating or working (I put a stool next to it so he can watch). If a guest was uncomfortable having a cat sit next to him at dinner, well, I'm sorry the Gato would survive one dinner from afar. If he tried to climb into the lap of my allergic friend (as cats often do), I'll scoop him up and take him into my lap.
If someone gave me the "it's his house, it's okay for him to annoy you" line for a person or a pet, I wouldn't come back. I understand that pets and children don't always know my boundaries, but I would expect a host to back me up on the "don't go through my bag, sit on me, or lick me" rules that I have for myself.