I think it was a good move for you to wait since your son was behaving well. Once he got cranky, it would be clear why you were leaving, and it would seem much more reasonable. Leaving earlier might have come across as passive-aggressive on its own--and, as you say, abrupt.
Annie's toddler may have been at a stage that is more flexible; or a child that doesn't have such a need for structure. And her child may simply be used to a very different schedule, if eating later is something that she normally does.
(when my first kid was really little, I told the doctor that I had her on a schedule. He said, "Oh?" Yeah, I said--I have a piece of paper, and everytime she does something, I write down what time it is.)