I don't find that especially complimentary, either. It reminds me of when somebody said to me, "I'd never have guessed you're homeschoolers. Your son is so well adjusted!" And the time somebody was shocked that my husband was a supporter of a certain political party, because "he's so nice!"
Or the classic, which I read in an Erma Bombeck essay, though I don't think it is original to her, "You sure don't sweat much for a fat girl."
When I pay a compliment, I try not to imply that the person has narrowly escaped being a member of a group I find odious.