Thanks for all the replies. I think, assuming they all came from North America, this means "pretty" does have a slightly different meaning for you as I really can't imagine calling a man pretty without the risk of his taking offence.
In my experience, it's not normally a thing you call the guy to his face. It's more a trait of how women talk to each other. I'm not sure exactly how that developed, but it's what I've observed.
Yeah, I probably wouldn't call a guy that to his face (although I hope most of the men I know wouldn't overly mind). But I wouldn't mean it in a derogatory way, if that makes sense. So maybe I shouldn't use it, but I don't mean it in a bad way and I just don't know how any particular guy would take it.
I wouldn't call a guy "pretty" to his face, unless I was confident that he wouldn't be offended. I do know a good number of guys who I wouldn't expect to be offended by being called "pretty." I don't think a guy's "prettiness" is derogatory or related to being effeminate.
That said, I generally don't call men "handsome" to their faces, either. I might make a comment like "You look nice/spiffy/snazzy today" when they're dressed up, comment on something they're wearing, or compliment a hairstyle, but I wouldn't be making general comments on their looks. If I did call a guy handsome or pretty to his face, it would probably be in a joking manner. E.g., "What's with the suit? Making yourself pretty for the job fair?" And that would be a know-your-audience thing--I can't think of anyone I'd make that joke to who would be offended by my using "pretty" instead of "handsome."
As Yvaine said, I'm most likely to describe a man as "pretty" when talking to another woman. This also tends to have a silly/jokey element for me. It's usually in the context of gushing over an attractive actor/character, e.g., "You're going to love this episode. It's full of pretty men.", "What was that? I was distracted by how pretty he is.", or (from my friend about John Barrowman) "He's so pretty, it's not even fair!". Less jokey descriptions tend to involve use "good-looking," "attractive," or "gorgeous." I don't personally use "handsome" very much, except in two contexts: (1) as shorthand for someone who meets our cultural standards of good looks but I don't personally find attractive (e.g., "[Actor]'s handsome I guess, but I don't see what all the fuss is about.") or (2) men with great gravitas, so that more casual terms don't seem to fit (E.g., Thorin Oakenshield
is a very handsome
dwarf; Fili and Kili