It's awfully hard for anyone to tell, at first glance, what ethnicity/nationality I might actually be, so this happens to me all the time. Until I open my mouth and fluent Spanish flies out (my family is Cuban; I was born in the U.S.). I don't really mind people speaking other languages (besides Spanish or English, that is) in front of me, nor do I really care - or assume that I am being talked about unless they are looking at me (in that way that tells you that people are talking about you). But being that my ethnicity/nationality is a little hard to pin down, I've noticed people refrain from doing this to me a lot.
The funniest thing that ever happened to me was when I was working as an assistant in the computer labs at the University Extension Center while I was in college. The Extension Center was where the foreign exchange students went to take classes, and one time, there was a group of young Argentine men looking at porn, which even in 1995ish, was against computer-lab policy. Of course, they were acting sketchy, but I knew they were up to something. I pretended to ignore it, until one of them said, "You think the lab assistant knows we're looking at nudie pictures?"
The reply from one of his buddies was, "That stupid little American? She doesn't have a clue."
I flew up from my workstation at the front of the room and said, in perfect Spanish, "Excuse me, but what do you gentlemen (HA!) think you are doing, and where do you get off calling me a stupid American?"
The stuttering response was, "We-we're sorry - we - we - we didn't know you spoke Spanish!"
My reply went something like, "I certainly do speak it, and I know you're looking at porn, and you and I all know that that is strictly against the rules. I'm going to take your names now, I'm reporting you to the dean of foreign student affairs, and I'm going to see to it that your privileges are suspended until you can learn to use the computers appropriately."
They apologized profusely, begged me not to go to the Dean, promised they would never do anything again that wasn't strictly study-related, etc. I stuck to my guns and told them no dice; what they had done was strictly against policy (there was no anti-gaming or anti-chat policy, but students who needed the computers for study got precedence - porn was a definite no-no, though) and they had insulted me on top of it.
I relented a bit and got their access privileges revoked for a week, and apparently they warned the other Spanish-speaking students that the lab proctor spoke Spanish, because I made an awful lot of friends from all parts of Latin America after that particular incident. Even the Brazilian students were pretty polite to me, because although I don't speak Portuguese, it's very similar to Spanish and I could make sense of it most of the time.
The Japanese students, on the other hand, were always unfailingly polite, didn't do anything that was against policy, and even helped me practice my Japanese, which I was studying at the time.
Not once did I ever hear anything that I might construe as an insult, even with my limited Japanese, from any of them.
