I think one thing that's going on here is that these people think they are hanging out with you.
If you had invited them over to spend a social evening together, it would be rude to laugh at a private joke and not share it. And it would be rude to make a phone call, especially without explaining something about it so that they didn't feel ignored.
But you don't get to choose whether they come into your store. So you are feeling that they're *not* "spending social time together." You think you're hanging out by yourself at the same place that they are hanging out.
Maybe you need to try to clarify that.
Either with them, or for yourself. If you're hanging out in *your store*, of course your customers are going to think that you are hanging out *with them*, because hospitality is part of what you're selling in your lounge area.
If it's friends and not customer "friends," maybe before you start reading or get absorbed in anything, say, "Well, I'm done visiting now, and I have some stuff to do, so I'm going to send you off."
When they ask "what stuff?" you should say, oh so patiently, "Stuff. And I need to concentrate, so I can't have visitors anymore."
With phone calls, maybe, "Oh, were you listening?" And stop.
When they say yes, then maybe "Oh, it was a private call. I'm sorry--I can't step into another office to take care of business." And again, stop--don't say another syllable.
If they persist, then say, "Please respect my privacy--I can't exactly go in the other room and shut the door, so I have to ask you to simply ignore things sometimes."