Well...if you aren't poor/disabled/a minority then to voice an opinion about something related to any of those groups would be taken less seriously. It's a "walk a mile in their shoes" type of thing. I can't speak to being a member of "x" group, so I'm not qualified to say how "x" group should feel about "y" issue. Yeah, I can voice my opinion, but it's not going to mean as much being someone who hasn't lived/experienced the issue firsthand.
I think the problem is that some people take it too far- because you're a part of a "privileged" group, none of your problems are as valid or worthy of concern.
I remember a discussion on some "social awareness" site, where the OP was lamenting the state of public transit in her area, namely a huge problem with sexual harassment. Offhand, she admitted to being what would be considered attractive, as if she had to apologize for being "pretty privileged".
The whole discussion went off the rails when others took it upon themselves to berate her for that, as if it were somehow a moral crime to be conventionally attractive. That and dismiss her concerns- that because she belonged to a "privileged" group, she had little to no right to complain about her problem.
In that mess, the real issue got brushed aside- that sexual harassment is a problem for everyone, regardless of how they look. And it can happen to everyone. And nobody should have to be afraid to use transit because they're afraid of being victimized by some foulmouthed neanderthal.