You know, I think everybody has some version of this to deal with. People just naturally make assumptions. It's part of getting to know one another. Especially if you're talking about the possibility of making a romantic connection.
So you're very tall - "Did you play basketball? What's the weather like up there?"
Very short - "Why do short girls always take all the tall guys?"
Very fat - "I lost a lot of weight on Jennie Craig; have you tried that?"
Very thin - "Are you anorexic?"
Black, white, yellow, red, brown, accent...something different from whatever you are...you name your question.
As for me, I was a military brat. Born in Spokane, Washington, USA. Lived there maybe a couple of months, I don't really know. Met my career military husband in the Philippines. I've lived all over the world, from Hong Kong to Germany, from Alaska to Panama. I've moved nearly 40 times in my life, never staying in one place more than a few years.
When people ask where I'm from, I never know what to say. If I tell them the truth, "Nowhere and everywhere - we're a military family." I have to listen to whatever their particular political prejudices are and they're often quite ignorant and judgmental and rude.
It's easy for us all to believe that we're the only ones dealing with these problems and be defensive and rude and smart-alecky right back.
But I've found that's never the way to go.
It's best to just say something innocuous from the getgo. Like, "I've moved a lot but I'm from here now and I really like the people around here, so I'm planning to stay."
And that's what I'd advise you...
"My family is originally from Lavenderland and I know I still have the accent, but we moved here when I was quite small, so I don't remember much about it. I'm from here now..."