OP here! Thanks for all the thoughts - I appreciate getting other perspectives.
To answer some of the questions raised:
1. As children we only had one set of Aunt/Uncle, who lived 12 hours away. They never attended any birthday parties and our family never attended any of my five cousins' birthday parties. I'm glad someone raised that question, because it does make me go "Hmmm!" now I realize consciously that this expectation is out of the norm for our family.
2. I am OK with the fact that DB and family don't travel down to us, because I do understand Busy Life, and because it's healthier for me not to resent it. However, there have been numerous times my DB was driving home from a convention (he's a vendor of specialty wares) and passed within 1/2 hour of my house, but didn't try to stay over, stop in, or even call to be met for dinner along his route. I know that sometimes he is probably just weary and wants to get home, but sometimes he's breaking his journey at a motel only an hour further down the road. I have found it hurtful but try to move past it.
2A. Ironically, 10-15 years ago, when our parents lived 6 hours from me, and 1 hour from my brother, I actually visited our parents more frequently. This is not true any more, as DB visits them at least twice a month to give them time with my nephew, and my health has cut my visits down.
3. We have invited DB and family for holidays and special occasions, but mostly stopped extending those invitations because they were never accepted. Last year we held a huge party for my husband's 60th birthday, and I made sure to invite DB, but they didn't come. I had no problem with that - it's a long way for a birthday party!
4. I think one thing that bothers me about this is, my brother knows that long-distance travel is difficult for me, with my health problems. If it's only going to be for a weekend, then my husband definitely has to come, to do the driving, and his schedule is often filled with work. If I'm driving myself, then I need to take several days, so there's at least a day of down time after the trip each way. So I wish my brother would understand it isn't easy to find the right time for a trip out to him.
Writing all of this out, I feel like I am making DB sound like a selfish jerk who expects all the flexibility and effort to come from me. I don't want you folks to think that! I love my DB and our relationship is pretty good. He does tend to let others do more of the work unless prodded, so I'm thinking this is an extension of that, with maybe some extra stress added in.
I know he is really busy this weekend, so I cannot call him until next week. But I will, and I hope we can talk about it. I want him to know that my not being able to come out is not a reflection of how much I love him or my nephew. I've resisted getting Skype (which he has suggested) but I now plan to get it so we can do video calls with my nephew. Schedules this summer are super difficult (both of us have many weekends claimed by work and hobbies) but hopefully we can figure out a date for me to come out and have a decent visit, even if it's this fall.