I think a lot of these statements, the one-timers not the people who repeatedly go on and on about it, are just thoughtless, careless remarks that people make because they've heard so many other people make them. Just something to say that doesn't require any original thinking, but you can still use the tone to express interest in the other person and their family. People around me say them all the time and I've said them myself, though thanks to eHell, I don't anymore, because I finally thought about what exactly I was implying.
I think I would just respond, with a big grin and display of affection, "Oh, we love our girls [or "kids"]! We're so glad they're healthy and they get along with each other well." Maybe give one a hug if she's standing there. I would say this even if it doesn't directly "address" whatever the person said, because they just said something silly and thoughtless and it doesn't deserve much consideration--I just want to counteract the bad impression my child might have gotten.
IIRC, you're not pregnant with #5 yet, but might be in the future, and there's a 50/50 chance of that baby being a boy, so I think I personally wouldn't want to say anything that might tip too far in either direction--like, I personally wouldn't want to say, "Oh, I don't know what I'd do with a boy!" because then what if I had a boy? You know someone would remember my off-hand remark and bring it up again.
Also, if all these remarks are coming from the same community (new church?), as you get closer to people, you might confide to a few of them, individually, that you feel hurt when people make those remarks, and you worry that your kids will feel less valued because of them. That might make a few people realize what is being implied by those remarks, and the sentiment might percolate through the community.