I'm going to disagree with the majority here. Yes, the happy couple can invite (and not invite) anyone they want, but by the same token, an invitation isn't a summons and if the OP's husband chooses not to attend because he has a problem with his children not being invited, he's well within his rights to do so.
Honestly, I understand his feelings. The 3 & 5 year old nephews probably aren't old enough to know or care about the wedding. An 8 & 12 year old on the other hand are old enough for their feelings to be hurt that they weren't invited to their uncle's important event, and as a father, I'd be hurt and insulted on their behalf... especially when one considers that we're not talking about a situation where inviting them would open up the floodgates and the couple getting married would feel obligated to invite a whole herd of kids.
Yes, the OP's husband not attending his brother's wedding can have relationship consequences, but by the same token, not inviting your brother's children to your wedding can also have consequences, like your brother deciding that his children aren't important to you or you don't care about them, and being offended by that.