I'm reminded of a quote from a book (I believe it was The Glass Lake by Maeve Binchy), where one character says to another "You look tired," and the other says something to the effect of "That's what you say when you mean 'You look old.'"
I've associated the two ever since reading that, whether or not the speaker means it that way. Sadly, it doesn't always stop me from saying it, but I'm getting better at keeping the phrase in.
I have to admit the thread title makes me think of the first place I heard it: the movie "
The Birdcage (35 seconds in)"
Armand: I didn't mean anything by it, I just meant you looked tired.
Albert: "You look tired" means "You look old" and "You look rested" means "You've had collagen."

Anyway, I can sympathize OP because I too have pale skin and relentless dark under my eyes.
However, I've never taken "you look tired" as an insult because I interpret it as a show of concern. People who know me have said it occasionally, not as critique on my physical appearance but because they are maybe noticing that I'm not my usual energy level, or slower to respond, or quieter in conversation...etc. I also have no poker face whatsoever. So they are wondering if I had a bad night, since for my personality that is more likely that being up all night partying.

If someone I wasn't as familiar with brought it up in conversation as OP described I would be irked because they were not close enough to me to be entitled to speculate on it.
No one looks good when they are tired, so it's not a good thing to say to someone as a general observation. IMHO pointing out people's seeming shortcomings, or otherwise speculating on aspects of their personal life (like how much sleep they get or what they do at night) without an invitation to do so, is impolite as a general rule. Instead people should stick to broader questions like "How are you?"