I used to live in a city where one particular street was pretty much lined with panhandlers and buskers. If I'd stopped and said, "I'm sorry" to every single one of them, I'd never be able to have a conversation with a friend while walking down that street.
One lone panhandler in broad daylight with lots of people around, I'll acknowledge with a, "No, I'm sorry," or some such. But in an area where there is one every five feet, or when there aren't a lot of people, or when it's nighttime. No way, no how.