I've gotten similar comments before - things like "You'll understand when you have your own kids" or "You're not a mom, that's why.."
My go-to response is to look at them and say calmly "You know, that could be very hurtful to say to someone who is unable to have children". They always look mortified and it shuts them down immediately. They don't need to know I CAN have kids, but choose not to - and I figure it's a lesson for them so that they don't hurt anyone who actually has been struggling with having kids.
Oh, this is one of my pet peeves at the moment. Recently I had someone tell me "You obviously don't have children. If you were a mother, you'd feel differently."
What an interesting assumption! I like your answer though.