People often use birthdays/funerals/weddings/reunions/holidays as a trigger to get together. Yes, we could get together ANY day, but having an event gives us something to focus around. For your brother, his nephew's birthday is that focus. Perhaps other people, like grandparents, travel for that event and it would be nice to have the whole family there.
However, for you, you a vacation locked in that weekend. So you have to ask yourself:
Would I like to visit some other time?
Do I see them at other times such as Christmas at another relatives house?
Is it easier for me to travel to them at this point as they have little kids and I do not?
How close do I want to maintain this relationship?
How much does the traveling affect my health?
This is all up to you.
You could decide that you want to make no effort to drive out to him.
You could decide to visit him, but at another time.
You could decide to encourage him to visit you (you drive to his hotel?) when he is nearby on his business trips.
POD to this. Different families work in different ways; and different people within the same family can have different expectations, too. I think most actions fall within the "not rude" range, but some can still be reasonably hurtful; and if someone isn't happy with their current situation, I think they need to step back, look at the big picture, and decide what they personally are willing to change (since we can't make other people change).
In her recent post the OP said that her brother sometimes needs to be prodded to step up and take on his full share of "work" in a situation. So indeed, maybe it's time to do that more. Everyone is busy, everyone has demands on their time or constraints on their travel, but the road still goes both ways. Instead of ceasing to invite brother to things--because he never came--maybe try stepping up the invitations, every month until he realizes you really do want to see him, and the only way that's going to happen is if he comes to you. Every time he bypasses you after a work trip, encourage him to stop by the next time. Not in a guilting way, but cheerful and upbeat and willing to help him overcome obstacles ("I'm sure you must be exhausted on your way back, we'll have the guest room ready for you and you can go straight to bed when you arrive! Then we'll go out to breakfast the next morning and chat.").
And the next time he does come, during a quiet moment I would lay out for him how difficult it is for you to drive long distances, and how much recovery time you need. Maybe you think he ought to understand this already, but IME people often think they've explained something well, when the other person really didn't get it.
Also, I like the idea of getting Skype (especially as brother suggested it) and trying to keep up with nephew more that way. I think sometimes it's more about the effort we put in to stay connected, than the specific way in which we do it, you know?
And honestly, if you try stepping things up for six months or so, and brother still won't visit and still complains, maybe then you'll know that his priorities are really different from yours, and there might not be a way to make him happy, so you should just do whatever feels right to you, even if he doesn't like it.