That's a fair point but I read the text as more "on my way", no apologies, as though he'd set off exactly when he intended to. I might be wrong there though, just because I have the slightly odd habit of always texting and telling people when I'm setting off doesn't mean everybody does. (I blame it on my parents making me continually late for everything ever while I was a kid until I was old enough to get places by myself)
Also I think he knew her original plan was three o'clock, if she stayed after 3 it was on his account because he asked her to and he needed to be considerate of that, which I don't think he was.