Coming at it from the point of view of the person ALWAYS expected to do the traveling to get together with family, it gets old, very quickly. We had family who came to see us 1 time in the 5 years we lived an hour away from them. During that time, we went up to see them at LEAST 5 times per YEAR. At that time, we had 3 young children and they had 2. So, if OP and her husband were always the ones doing the traveling, and her brother, even pre family, was not, I can see how the brother gets locked into the mind set that he stays put and she comes to him. And she (OK, me...) starts to resent the daylights out of it.
After a while, you aren't willing to BE the one doing the driving all the time. Even more so if you're 7 hours one way from the party. That's an overnighter, in the middle of a vacation with other people. Not a reasonable expectation on the part of the brother. Brother needs to learn that the road goes both directions.