OP, there was no way you were rude. I think I would have been uncomfortable in that situation as well.
My late MIL once answered the door to a salesman who was giving away free baking dishes (similar to Corning Ware) if you listened to his speech. This was the late 1950s/early 1960s. She politely declined taking the dish through the screen door but the salesman tried opening the door and insisted she take the dish. She grabbed my FIL's hunting rifle that was next to the door and pointed it at the guy and said "I said no thank you". She told me she could hear him running all the way down the road. I really miss her sometimes.
I love that!
When we first moved here 3 years ago we had people trying to convince us to go to their church. None of them were denominations I was interested in attending simply because, well I know what I like in a service and I knew they didn't do it that way and the church I did attend and the one I do now aren't door to door churches. Some people were content to just give me their pamphlet and move on while others didn't want to take no for an answer.
Anyway one guy was particularly persistent. "Come on out to our church, it's good for the kids!"
"No thanks, we don't attend church" (we didn't at the time)
"You must, you need to, don't you boys (my kids had been playing on the porch) want to have fun in church?"
"Mom can we please go!" I looked at the guy and said "No, we don't go to church, and we will not be going to yours."
The guy said "Well we (we being him and another guy) have a van here so you can stay home and we'll pick your boys up and take them to church with us!"
Now let me tell you that sent warning bells off in my head and I told him absolutely not, my children do not go off in vans to churches I don't attend with men I don't know, thank you. They wouldn't even go off to a church I don't attend with women I don't know. I still shudder when I think of it.