"I'm afraid it won't be possible" seems like a remarkably polite way to respond to someone who drops off an empty envelope and comes back in person to collect it, expecting it to be full of money. To my mind, it takes a great, big brass set of...well, you know...to even leave the envelope in the first place.
"I'm afraid it won't be possible" would be a perfectly good response, as would "Sorry, I've already donated" or even "I prefer not to give to that charity", as long as they're said politely.
From the other side: My Mum used to deliver and collect those envelopes on behalf of Christian Aid and I went with her a few times (never collected the envelopes myself, Mum was the authorised collector so it had to be her knocking on every door - I was just there to keep her company). The door to door collection is organised through local churches and every collector is a volunteer. My Mum definitely does not have a great, big brass set of anything; she's actually incredibly shy - but it's a cause that she supports and is willing to put herself out in order to support. Nobody in our church expected everyone to donate; in fact the general assumption was that most people probably wouldn't donate - that was their choice, but everyone got thanked for taking the time to open their door to us.
I actually prefer to receive the envelope through the door: it gives me time to (a) decide if I want to donate anything and if so, (b) exactly how much. The alternative would be someone on my doorstep with a collection can or selling raffle tickets. For some reason, I always feel more obliged to give when they rattle the can under my nose - maybe it's the element of surprise.
Sparkle Star - yes, I've known some folk helping with Christian Aid collections who will appear to make a mental note of who's given and who hasn't, I think every church has that type of person

. They're not worth worrying about. Other collectors, like my Mum, will just be grateful that you opened the door; doesn't matter whether you donate or not as long as you're polite. Actually making a donation is a boost, but most of the people doing these collections are reasonable people and they understand that not everyone will want to support a particular cause.