I am that trivia whiz. There are people in my circle of friends who refuse to play Trivial Pursuits because of "how easy it is for Cwm to win". Except I've lost the last five games in a row I've played. Yes, it's been close, but I have bad luck with the cards and always get the ridiculously hard ones, and other people get the ones that are quite easy.
It actually hurts, having people tell me to my face that they refuse to play a game if I'm playing. I don't like being a guinea pig on games, which is what usually ends up happening. Someone has a new game on the scale of Arkham Horror or Game of Thrones and we spend four hours looking up rules and clarifying things online because the fifty page rulebook isn't clear, and I can't stand it. I bow out so I don't end up throwing a fit (I know my limits), but people don't want to play the "boring" games like Clue or Monopoly or Scrabble (actual board game). They won't play trivia with me, I won't play long drawn-out games, and that leaves us with about three games that we can agree on, which everyone is getting tired of.
It leaves me out of things. When everyone else in the group starts clamoring when someone suggests trivia because I have an unfair advantage, they all then decide on a game that I'm not likely to play. Nearly everyone in the group knows this. I have a reputation for not wanting to play those games. So I end up sitting aside, hurt because I'm "too good" with trivia. They all have fun, and then tease me for not wanting to join them in the fun, which hurts even more. There is no winning at this point.