I have two different answers for two different situations.
At work: You just say, "Please speak English." It's fine. They know that it's the common language, so you don't need to explain anything. Think of it like asking someone to speak up a bit if there is a lot of background noise or they are whispering. You aren't criticizing them, you're just asking them to speak in a way that will be understood by all.
In a social situation, assume that they aren't aware of the fact that you aren't following along. Say, "I'm sorry, but I don't speak 'otherese'." If they invited you to join them and you aren't following along, of course they want to know! (However, if you asked to join a different group that was already established, I don't think you can politely ask to be included.)