This is a bit of an old one, and it generally doesn't bother me as such. However, I do find it off-putting when people assume that "of course, you wouldn't know about that".
For instance, when those older than me assume I couldn't possibly know how hard it is to deal with elder care. Being born to older parents, and having married a husband also born to older parents (all of whom are now deceased), I have actually dealt with a LOT of elder care, from the age of 26. So, yes, I do actually "know", from a very personal viewpoint (especially as three of the said parents had severe debilities for many years). As mentioned, it generally doesn't bother me, but at times, those saying such things have said it in such a way as to be quite a put-down.