Definitely not rude. Trust your instincts. You didn't feel safe and you had every right to close the door on his face. He might've been the sweetest guy ever but it's not rude to protect yourself, esp given the fact that he is intruding on your personal territory.
I once refused to open the door for a salesman and he kicked the door and said "open it." Yeah, as if that was going to all of a sudden convince me I was wrong for not opening it up. 
To the OP, absolutely no, you are not being rude. You felt unsafe. You did the right thing.
Roe: I have a "don't open the door" policy, too. I am home alone all day. I am cop's wife. Over the last year or so, there has been a dramatic increase in daytime burglaries in our town. If the doorbell rings and it's a man I don't know, I don't open the door. I have a card I hold up to the window that says in big bold print, "Thank you, but I am not interested." I have had some men yell, "That's so RUDE!" or "Come on! Just open the door." Er, no, thank you.
A few weeks ago, I was outside getting my mail and a woman with a clipboard caught up to me before I could get to the door. She wasn't wearing a uniform. All she had was a blue piece of paper that looked like an advertisement for a carpet cleaning special, but when you looked closer at that paper, there was no business name. There was no logo. During her spiel, telling me that "a new carpet cleaning company" was running a special for the holidays, she never said the company's name or gave me her name. There was no official paperwork on her clipboard, just a (handwritten) list of addresses. She was REALLY pushing for me to let her in so she could look at the carpets and give me an estimate. I said no. Repeatedly. She didn't give me a card, "just in case I changed my mind," which is standard operating procedure for door to door sales. That REALLY set off my hinky meter. She just wasn't doing any of the things you expected a salesperson to do. I am pretty sure she and the person in the unmarked blue van down the street were sizing up houses for break-ins.
After a while, she noticed I wasn't moving toward my door. She said, "Aren't you going inside?"
I said, "Not until you leave."
She said, "You don't have to be RUDE about it!" and huffed off. Before calling all of my neighbors to tell them not to open the door to this woman or her partner, I called the police non-emergency number to report her behavior as suspicious and give a description. The dispatcher told me there had been similar reports in other neighborhoods. So far, we haven't had any break-ins.