Years ago, I worked as a receptionist in a medical office that was doing pre-exam physicals that had to be passed before the candidate could sit for the correction officers exam. It was a very tough physical exam, which included running on the treadmill & having to lift heavy weights. I have never met a more impressive, professional group of people. I can recall saying that it was a job I'd never want to do, but very grateful that these folks did.
Considering the reaction you're getting, going forward, I would either steer the conversation so that you aren't asked the question or be vague in my answer. If pressed, I would answer in a tone that was positive & upbeat as this usually keeps the disparaging remarks at bay. I guess disparaging remarks, I would try to shrug off & then bean dip. You don't want to get into debates with customers.
Good luck & I hope it all works out.