I don't think you should just back out of the wedding.
Does your brother and his fiancee live close to you?
What I would do is go over when you know they are both home and talk to both of them at the same time.
This way, the fiancee' cannot say that you said anything you did not say.
Start out, "Mike, I wanted to come over and talk to both of you because there seems to be some confusion and I do not want to start your wedding off with hurt feelings. Mindy, as far as the b-party, here is the deal. I had already planned and booked my vacation way before you booked your party. I am sorry, but since you changed your venue, I simply cannot make it. It has nothing to do with you personally. I would have loved to attend and I hope you have a great time!"
You stated in your text that you have been hearing things. Can we discuss those things now so that I can dispell any myths or untruths that someone is apparently spreading? I am here to talk to you in person and I hope that as my future SIL that we can always discuss things between ourselves and not automatically believe what someone else is saying.
Put the ball in her court in front of your brother. This way, she cannot say that you did this or that, because he can always say, "well, did you hear it from her mouth? Remember that talk way back when?"