mmswm, this may take some time, but I think you need to work on not smiling. A person who is not smiling is not mean or rude. They are simply a person who is not smiling. You do not have to be smiling all the time for people to think you are nice.
In fact, by smiling all the time, even when other people are being mean and rude to you and your children, you are doing them a disservice. By smiling, you allow them to think that their actions are correct, that they have a right to give you dirty looks and make mean comments about you.
When your children misbehave, do you keep smiling at them and using a pleasant tone of voice? Or do you stop smiling, gradually use a sterner tone of voice, and by other body language clues, show them that they are beginning to tread on thin ice?
Start training all these random bystanders that they are in the wrong. Well, they are probably not going to come to the realization that they were wrong based on one encounter with you. But don't train them that it's okay to make rude remarks to strangers. When they give you a dirty look, try shooting a stern look right back at them. Don't be nice and pleasant. You don' t have to be nice and pleasant to avoid being rude. You can be blunt and a bit stern while still being perfectly polite.
My SIL, who has a 13 year old son in a wheelchair, is a tiny little petite woman of about 5 feet. When she gets harassed by bystanders about the handicapped placard or where she's parked, her body language is not defensive. Instead, she stands tall (well, as tall as she can), she looks the other person right in the eye, and says, "My son needs the handicapped access," in a strong, firm, authoritative voice. Then she breaks eye contact, and turns back to what she was doing, dismissing the other person completely. She takes control of the situation; she doesn't let the situation control her. And most of the time, the other person slinks away, aware that they have crossed a line they shouldn't have.
Think of it as teaching your children to stand up for themselves.
I'm sorry you have to deal with this. It sounds like you have enough to deal with as it is.