Same, I'm self-employed and I have flexibility as to when to make other appointments, or take half a day off, and so on, but then everyone seems to assume that I can automatically take time off to do whatever. So if I'm joining a weekend trip (Saturday/Sunday), it'll suddenly be announced we're leaving Friday evening to get the driving out of the way (fine) but as the time draws closer, suddenly I'm asked to leave work at 4, or 3, or even 2 PM, when that wasn't part of the original invite. Uh....well, I am my own boss and can do what I want really, but it's not in my best interests to turn away business on Friday afternoon so I can do a fun trip. It's money. And I need money.
I can totally see someone thinking that because I can be flexible about deciding when to have lunch, then I must have all the time in the world to hang around waiting if they don't show up at that time.