This is not as unusual as you might think. Basically the decision comes down to this: Can you in good faith stand up and be his best man and support him as a friend?
Many people will tell you to only do it if you support the union. However, it is possible to be supportive of it and a good best man whilst still disliking the bride. I know of one scenario where the best man pulled out (and unfortunately the bridesmaid who was his fiance!) because they decided they didn't want to be part of it due to not thinking marriage was the right step for the B&G. The couple did have their issues, but I'm pleased to say they have worked through them and a few years on they now have a very happy marriage and two beautiful children. The friendship with the BM and bridesmaid who pulled out was irreparably damaged. The "replacement" best man also had some misgivings but supported his friend regardless, and their friendship is stronger than ever. I know of other situations where the best man and friends have supported the relationship despite misgivings, and then been there to pick up the pieces afterwards when the relationship later fell apart as predicted. What I'm saying is that you never know what will happen in the future. You might be wrong about the bride, you might be right. But now is when you have to decide whether or not you will be supportive of your friend.
Either way, I think it's appropriate to gently explain to him as a friend that you have misgivings about the relationship, but that you want to be a good friend to him. Do so in a non-confrontational way and hopefully he'll be open to it. But I think you know that you're not going to change his mind about it either way, so it's really your call as to how far you support it for his sake. In my observation, friends pulling away will often push a person further into a bad relationship, having the opposite effect to what they intended. So that's something else to take into consideration - if he's marrying somebody toxic, perhaps he'll need your friendship more than ever. If nothing else, for support when it all goes wrong.