I do understand your feelings. Even if the goal of the trip is to get the cousins together, if BIL by himself makes an already uncomfortable situation worse, then I too would be dreading not having my sister around to soften that a little bit.
But, I think your opportunity to clarify the invitation has passed. I could have been on board with you clarifying that the invitation was for all three of you in the moment, but to do it now, is insulting to your BIL. And, like you said, you don't really get along with either of them, so the point really is about the cousins...your sister and BIL seem to understand that, so the goal is still being met. Because you're willing to do these vacations, despite your relationship with the family, in order to cultivate the relationship between your daughter and her cousin, then it seems to me that maybe the fallout of rescinding the invitation (even in the moment) may not be worth the risk to your daughter.
My advice is to take this time to focus on how you would deal with potential conflict if it were to occur. Cousin is hard to handle without mom around? Ask your sister for tips for the times when you'd be watching the kids alone. Worried about not getting time to yourself or date night with DH? Then work out the schedule of family and alone events ahead of time. BIL driving you crazy and you need to step away? Work out some code words with DH where you can tag team and both take moments to breathe before reengaging. And, maybe your sister will go afterall, or they will decide that none of them can make it, or it won't be as bad as you fear.
Also, as the girls age and become easier to handle, you might consider (if you want to of course) offering to take just your neice on vacation with you. Word the invitation like "DD would love to spend the time with neice. If you two are ok with it, can we bring her to DW with us? It would give you some time together, too." And maybe they'll return the favor and that will be a new system that works better for everyone.