I am one of those people that must answer a phone during certain times. I am occasionally the on-call for a nearby facility and while it is rare for me to get a call, I absolutely must answer the phone if it rings. I do try to warn my friends and family if it's my week, just in case they'd rather not go out for dinner (go to the game, have me over, etc.) when I might be interrupted. So far nobody has declined and everyone is pretty understanding, even though the usual call results in me needing to leave wherever I am on very short notice.
I think as long as you keep the conversations short and make your face-to-face companions aware that they are as important to you as the phone call, that you're OK.