First - you did nothing wrong.
Second - As a person who does not care for kids (and I hope that I do not have to explain to the folks remaining on this board that I do not HATE them, I just don't care for them...but anyway), but loves many, I appreciate ANY parent who knows what their kids' limits are and knows that they cannot generally be expected to suck things up like adults. Kids are DIFFERENT and have different needs than adults. Although I could be persuaded to partake in nap time again, if allowed...
If my godchildren are tired and cranky when we are out and about, home we go - even though I might have wanted to stay longer (and even if they wanted to stay longer).
I think that it's awful that your family doesn't understand. Unfortunately, they won't understand until they choose to. They are seeing things only in terms of themselves and how the issues you deal with every day are interloping into THEIR wants. That's their problem. While I am not a big fan of "the chiiiiillldren come first" - in this case he does. Because he has to and you are being more than reasonable.
Of course, the change in plans you propose would really be no skin off my nose (and I judge many things by the "skin off my nose" test).
Therefore, I automatically place the problem squarely on them. Do what's best and if people don't understand, then you can't help them.