I got this all the time when I was taking my elderly father to the store, to doctor's offices, etc. He had a series of strokes and could only walk with the help of a walker, and that was on good days. Other days, we used the wheelchair. So, naturally, I got a handicapped placard which I used only when I took him places. Yes, I could have let him out, driven off to park and walked back to him. But he got very anxious and panicked when left alone, even for a few minutes.
When strangers came up and politely reminded me that I parked in the handicapped spot, I explained about my father and they always apologized and wished me well. If they gave me grief, I just gave them a cold stare and asked them why they enjoyed terrorizing the elderly. They would get it when I went around to the trunk, took out the walker or wheelchair, helped him out of the car and slowly --- very, very slowly --- went on our way. A few mumbled apologies, but most just slunk off.
This is a two-edged sword. I'm glad that much of the general public feels that the able-bodied who exploit handicapped parking solely for their own convenience should be called out. I've been out with my dad when all the handicapped parking was taken, and some of it by people who obviously didn't have disabilities and had neither a disability plate or placard. (Yes, I know that many disabilities are invisible, but when you have school aged jocks wearing their letters and bouncing along, I assume any disabilities they have lie solely in the moral realm.) On the other hand, people should realize that the disabled person is not necessarily the driver and hold back any comments until they are reasonable certain that no one in the car has a disability.