I am that trivia whiz. There are people in my circle of friends who refuse to play Trivial Pursuits because of "how easy it is for Cwm to win". Except I've lost the last five games in a row I've played. Yes, it's been close, but I have bad luck with the cards and always get the ridiculously hard ones, and other people get the ones that are quite easy.
It actually hurts, having people tell me to my face that they refuse to play a game if I'm playing. I don't like being a guinea pig on games, which is what usually ends up happening. Someone has a new game on the scale of Arkham Horror or Game of Thrones and we spend four hours looking up rules and clarifying things online because the fifty page rulebook isn't clear, and I can't stand it. I bow out so I don't end up throwing a fit (I know my limits), but people don't want to play the "boring" games like Clue or Monopoly or Scrabble (actual board game). They won't play trivia with me, I won't play long drawn-out games, and that leaves us with about three games that we can agree on, which everyone is getting tired of.
It leaves me out of things. When everyone else in the group starts clamoring when someone suggests trivia because I have an unfair advantage, they all then decide on a game that I'm not likely to play. Nearly everyone in the group knows this. I have a reputation for not wanting to play those games. So I end up sitting aside, hurt because I'm "too good" with trivia. They all have fun, and then tease me for not wanting to join them in the fun, which hurts even more. There is no winning at this point.
Also, there is such a great amount of expectation that a person has to live up to in that case. People expect you to be the best at a particular thing, and when you fall short, they somehow don't like it....even though your being better at it than they are is something they don't like either. Especially if you are on a team and your team ends up not winning. If you're considered the best at this particular thing, your own team is mad at you for letting them down, and the other team teases you in a way they wouldn't if they considered you to be only as good as they are.
What I'm thinking of in myself is not exactly a competitive sport, but because people consider me to be good at singing, I'm honestly uncomfortable even singing "Happy Birthday". Because I get stared at. I don't know what people expect but it seems I can never figure out what that is, and meet it.
I think this has a broader application, though, in that when one person is a lot better at the activity than the others, those others are always going to have some unrealistic thoughts that the one who is better has no control over, no matter what they do. They are stuck between the rock and hard place of being perceived as arrogant, and being perceived as having false modesty. Do you minimize your ability and risk being told you are "fishing for compliments"? Do you acknowledge your ability, offer to take a handicap or bow out, and risk being perceived as "showing off"? Where is the line to balance on, where everybody else will be comfortable and no one will take any offense?
It's impossible to know. Which is sad, because then the activity you enjoy (so much, that you have become just that good at it, and take pride in your accomplishments regarding) is now something you can only do as though it is work, never again as an informal activity with your friends.
My vote for the situation in the OP? I agree with those who say pick another game. One everybody likes and has some element of chance. Bunco and Yahtzee are good candidates for this, IMHO.
And everybody also should keep in mind what others have suggested in terms of the question of "Why are you playing in the first place?". It's probably a question that could use some explicit discussion before the games, sometimes, I think.