I don't respond to panhandlers either. But then again, I don't make eye contact with or respond to those people selling stuff at kiosks in the hallways of the mall. A request for my money is not a social situation. If a panhandler sneezes, I'll say "bless you", but I won't typically give them any money. I've been known to let out bloodcurdling screams when surprised by people walking up to my vehicle as I'm strapping the kids into their carseats. That's a startle reflex, I don't just scream when people walk near me. But if I turn around and there is suddenly, and unexpectedly, a person in my personal space, it freaks me out. Especially since I'm a 5'3" woman who is usually the only adult with 2-4 children under age 3.
DH is a sucker for a story, especially if there's a child and a picture involved. At a nearby big box store, there is a family that occasionally walks around asking for money to help pay for a child's cancer treatment. They seem pretty knowledgeable and will talk about the tests and procedures that are taking place, and they always thank you for your time. DH gives them what he can, whem he can and comes away feeling grateful that our family is healthy. I think it's kind of hinkey because they also usually have a photo of the child, and will tell you a bit about the sick child, but I've seen several pictures of different children. Boys, girls, curly hair, dark hair, blonde, freckles, no freckles, etc.
I've also had the unsettling experience of being boxed in and yelled at by someone asking for money. I was backing out of a parking lot and he positioned himself in such a way that I could not continue without running him down. Again, it was just me, my 3yo and my infant daughter. He yelled at me, and when I told him I don't carry cash with me, he demanded that I go into the store and get some for him, all the while he was calling me all sorts of nasty names and telling me that I was making him feel so bad and ashamed, and didn't I know how embarrassing and demeaning it was for him to have to ask for money. I had to call 411 to get the store phone number, call a manager and blare the radio (drowning out the obscenities before my little sponge in the backseat could soak them up) while I waited for store personnel to come rescue me.
OTOH, I've also had a grown men weep when I turned my car around to stop and give him a sandwich, powerade and banana. And I had a warm, fuzzy moment when I was able to give a woman with a bunch of little kids about a dozen pouches of tuna and some random groceries I had in the car. The kids' eyes lit up, and it broke my heart a little.
(I find that being inside a locked vehicle, with DH for company makes me more willing to talk to/help those who ask)