I don't think it's necessary to avoid telling people what you do, unless this sour reaction is really wide-spread. It can be good to chat with someone about your profession if they're not biased about it.
I had a friend who was a debt collector, actually, and then switched fields and worked in taxation. You'd have thought he'd be tarred and feathered. But actually, he was really good about it. Whenever someone started giving him the stinkeye about his profession, he'd say "it's a lot different to what people expect. Why, just the other day..." and tell some hilarious anecdote about his work that was unexpected and bizarre enough to have people interested or at least thrown off-guard, and we could all laugh together about how weird the situation he described was. And nothing bonds people like laughing together. He had a real corker about a woman who tried to pay a $15 fee using a cat that was supposed to be some fancy breed--she turned up at the office, put it on his desk, and walked out before he could respond. It's strange enough that people always start asking 'what was she thinking?' and 'was she crazy?' instead of focusing on how he was trying to apparently commit ethical sins against her. I'm a teacher, and am often informed that I'm the cause of every ill in this world, but have used his technique with more success than I expected, despite being nowhere near as charismatic.
Anyway, if you have any good funny stories from your work, they might help mellow things out a bit. Worst-case scenario, if you see one of the 'types' of people who you've noticed tend to have bad things to say about what you do, just be vague to them specifically, or bean dip and never let them find out at all.