Well, if that is what he thinks, isn't it better to know so that you can address it? Either he's got a point, and you step up what you're contributing, or he's missing some stuff you're already doing and you can talk about that. The alternative, if he thinks you're slacking off in some way, is that you have no idea and he quietly resents you until it's an actual problem.
In my experience, there are few things in relationships between people (but especially romantic, cohabiting relationships) that are best left hidden, unsaid, or buried.
The other thing I noticed, in going back and reading your original post and the one you just made, perhaps part of the problem with the joking is that it's kind of competitive. Competitive joking can be fun, if everyone is in the right mood, but if the desire to "win" is strong, it can lead to bad feelings at the end.
In addition to that, the amount of work one puts into a relationship/household shouldn't really be a matter of competition. Going down that path does not instantly lead to ruin, but it can set up some pretty bad precedents. When you start comparing who works longer, who earns more, who washes the dishes more often, etc., and using those things to decide on a "winner" (and therefore a "loser"), things can quickly deteriorate into a mess of bad feelings. It's important to compare stuff like that in order to determine a fair distribution of household work, but once you've distributed everything, it's not especially helpful to focus on it. Better, I think, for both people to acknowledge the hard work the other person does, and express appreciation for their having done it.