The reason its so bad, as demonstrated by the people who are hurt by it, is the statement is a "you" statement not a "me" statement. The person saying it might mean "I'm having a rough day" or "I miss the freedom I had before I had kids" or "I wish [for a moment] my life was different". But they aren't saying that, or anything about themselves. They are making a "you" comment - "lucky you" - without knowing enough about that person's life to really make that kind of comment. That person might not be "lucky" for not having kids. It might be due to diligent care on their part as far as contraceptives, or it might be due to fate or tragedy, or all sorts of reasons.