If it were something I felt obligated to attend--like a farewell party--but not something I enjoyed, like a game of some 4-person shooter on Xbox, I would be very happy if my friend said to me beforehand "Look, I know you don't like Rip Roarin' Shootin' 4, would you like to do something for my goodbye, just the two of us? I know that Mr Jones loves the game, I'd still like to have him over as our fourth, but I thought you would prefer if I met with you for lunch at XYZ Cafe the next day." I wouldn't be offended and, in fact, would be relieved to not be the person sitting and staring incomprehensively at the screen (or in OP's case, the board game), trying to start conversation and feel included when I'm not interested in the least in the thing everyone else focuses their attention on.