My cousin had something similar happen to her, and in her case, it WAS rude. Her ex BF had partial season tickets to their favorite major league team. So the tickets were his, but they would always go together. One day they're at a game, and he says he's going to go say hello to a friend sitting elsewhere, who cousin couldn't stand. She was fine with him going to say hi, but apparently he stayed, had a beer and was gone for more than an hour, leaving her alone.
Even though I couldn't stand her ex, I thought it was rude on his part. To make matters worse, they had to clear the stadium due to t-storms, while he was off "visiting" and he wasn't answering his phone, and she had no clue where he was. Plus he had the car keys so she couldn't have left (which she said she wanted to do) even if she wanted to!
But a quick pop by to see someone, say hi, and then come back, esp when Aunt had others she could socialize with, is fine.