OP, your ex is being manipulative. It is not your job to comfort him about this. The fact that you did so is because you are a good person, but he is showing his true colours by trying to make you feel bad and then refusing to give you the facts.
Stay away, stay far far away. Don't ignore him as that gives him ammunition against you, but be coolly polite. Almost too polite even - charmingly, arms length polite. And do not let him break down that barrier under any circumstance. Keep your distance.
Re the rumours, ignore them. If somebody says something to you personally, then behave slightly shocked, confused and refute their assertion then bean dip. Otherwise, pretend that the rumours don't exist and be your confident, lovely self. People who meet you will assume the rumours are untrue when they see what a lovely person you are. That's the nature of people. If you act ashamed or as if there's something hide then they'll believe that there is. Good luck, and ultimately believe the following: This Too Shall Pass! :-)