I'm in the UK. Amongst everyone I know, whether its a formal or informal dinner, 'come over for dinner' means come and spend the evening with us. Unless it's specified otherwise, e.g. come over for dinner before we head over to the movie, something like that. Same with Sunday lunch, they would probably end up staying the whole afternoon.
Amongst my friends, and also observing my parents friends as I was growing up, guests come over a little earlier than dinner. If it's informal, more like, 'come share my food' then guests often end up offering to help, or at least having a drink in the kitchen with the host, or sitting in the living room with others if there's a number of guests. Then you eat, then you move back to the comfy seats and watch a movie, play a game, chat, share a drink. The only difference if it's a formal dinner would be on the level of helping out and the type of food served really.
Of course if it's a week night people will start leaving around 9.45-10pm and it could be anytime if it's a weekend.
If someone came only to eat and then left it would be considered rude, like you didn't actually want to be with the host/other guests. Unless, again, specified in advance - maybe the guest says they have an early start etc. But they really should say in advance.