With family, I always took it to mean, "Come over, interact while dinner is being prepared or eat as soon as you get here, socialize over dinner, have coffee and/or dessert, interact some more but don't overstay your welcome." I'm not a very social person myself, and I like to know (or have a basic idea of) how the event is to play out beforehand. So, if dinner was several hours later than they said it was, there were very little or no snacks during a long visit (though snacks aren't a requirement), and conversation died before the visit ended, I'd probably be irked if I had to continue to stay. It's one reason I prefer to drive myself rather than go with the family if I can get away with it, because I can only handle socializing in small doses. I don't know if it's rude or not to leave a bit early, though. I'm in the US, and I pretty much take my cues from what everyone else is doing.