OP, I sympathize with your desire not to sing, as I am blessed with a singing ability which could turn "festive caroling around the piano" into "shock and awe, where are the ear mufflers..." Plus, I too dislike having celebratory activity participation forced, rather than willingly joined, or not.
That said, in your shoes, I would find an inexpensive musical triangle (Amazon has some for $3-7), or a small bell and plan the optimal, and enthusiastic use of that *as* my participation. Christmas carols can benefit from punctuation, and you can even mention that you are incorporating the famous line from "It's a Wonderful Life" and helping angels get their wings. Or you are going for a Sintra "ring a ding ding" style - at any rate, you would not be refusing to participate, but choosing your form and means of participation.