I POD dawbs. Don't explain. Don't respond. Don't say, "You must think poorly of my relationship if you think me having a male friend is a threat to it"? They aren't making these comments because they think poorly of your relationship. They aren't thinking at all.
Back in January, my husband was "hosting" a female blues musician who was visiting Oahu. She was traveling with a male friend who ran the marathon. As I got to know her better, I discovered that she was married, her husband was in the military, and he doesn't travel with her very often. Later I mentioned to my husband that they must have a strong marriage if she can travel with a male friend. DH looked at me a bit oddly, then said, "You realize that [male friend] is a *man who loves men.*" Um, no. I didn't realize that. Mostly because I don't think about casual acquaintance's preference in scrabble partners.
I still think they have a strong marriage, but what do I know? Not much more than Mental Magpie's thoughtless (& humorless) coworkers.