Yes, some people do not grasp sarcasm or nuances or anything other than literal in speech. Typically, I cannot be particularly close with them because it is so exhausting for both of us!
I was friends with a few of these in college, but only because I was away from home and not really with "my people." I hung out with them because they were available as companions, but I never really felt we were ever really on the same page, humor-wise. It really got a little tiring to be given a blank, curious stare, a concerned look, or a comment, "I can't believe you're so worried about that" in response to some tongue-in-cheek comment of mine. Also, I don't think my sense of humor is particularly odd, because other people seem to get it. It was just that particular small group that seemed to only ever take a literal interpretation of everything. Which was why I never really felt a true connection with them and didn't keep in touch after we all went our separate ways after college.
I take the OP's comment as tongue-in-cheek, good-natured teasing. I'd want to say something like, 'Oh please, so you didn't get that I was joking, that's fine, but feel free to drop it any time."
As for the reading, I'd be too embarrassed to admit I don't read much. Actually, I don't read much (I used to) because I find myself busy, or lazy, or distracted by the internet, but I'd never come out and SAY that on FB. Somebody posted a list of 50 classics with "how many have YOU read?" and I was too embarrassed by my answer (6) to post it. Instead I resolved to try and read more. But to actually brag about not reading or disparage someone who does? Meh, says more about the person saying it than the person doing the reading.