I've first got to say that, in my ethnic cultural heritage, we would never dream of dictating to an older generation relative when he/she gets to arrive/leave. It's just not done and is considered rude and disrespectful. Coming from that viewpoint, everyone would be expected to pitch in and drive the relative around if asked -- well, at least, the relative in my mind is my beloved grandmother.
While I understand your standpoint, it still wouldn't excuse grandpa to lay down the times. He could easily ask of course, but if it's inconvenient or impossible to the driver (the drivers with the newborn for instance can't just adjust a baby's schedule), Grandpa should be willing to adjust his times. If he doesn't want to do that, fine, but he can then call around to make other arrangements. Worst case, he grabs a cab.
But to treat family like a taxi service with demands on 'pick me up then and drop me off NOW', sorry, that is rude. Being an older generation relative does not excuse that. You're family, so you're more than able to talk and discuss, and find a solution to benefit everybody.
OP, not dissing your grandpa here of course, merely using as example

While I agree that alternating turns would be best in this case, I hardly see the problem if one or two people don't want to do it. As long as you have four people helping out in this, it's still an easy ride.
My husbands family does the same for their grandma. She has nine kids, but only two or three ever drive her to these gatherings. We never really make a fuss about people 'skipping turns'. She generally calls around herself until she has a ride. Most of the time her kids beat her to it, and call themselves to offer.
If she gets tired early, and it inconveniences the current driver to leave right then and there, there's always somebody else that pipes up and goes to drop her off. Or grandma waits another five minutes and leaves with the next party.
So it mostly depends on the when and where and on family situations who drives. To them, it's not a chore. And it's really not worth getting into a family feud over.