I had something like this happen to me recently....where I felt belittled but didn't want to create a huge drama. I had posted on FB for book recommendations and I got a response from a family member: "Don't you have anything else to do other than read?" I found it pretty insulting, and no, it wasn't a joke. This person has probably never read a full length book in her life and I guess she thinks it's a waste of time. Did I mention the book recommendations were for a vacation, meaning I wasn't supposed to be working or anything else during that time? I ended up responding that reading was a luxury for me and exactly what I wanted to do while on vacation. Ugh.
I was actually just going to post about what to say when it
isn't a joke. The suggestions I got were great BTW - thanks everyone for sending them - they are non-angry and matter of fact and I will be sure to try them out when dealing with those people who take things a little too literally

Miranova has a good point though.....often when it is is a putdown we can't say "you misunderstood my joke"....in fact I remember being told the same thing - I love to read and it is pretty much my favourite "downtime" thing to do.....and someone had to say in a bemused tone "Wow you must have an awful lot of time on your hands"
Of course I got defensive : "Hey I work full-time, I have two young kids, I spend lots of time with family and friends, I hike, I ski etc"....red-faced and feeling like an idiot for trying to justify myself.
It makes me wonder why some people feel they can say stuff like that to others. I used to just sit silently and seethe because I was brought up in a "don't make waves" kind of house but now I would rather politely shut it down.