My mom and I go to about 8 baseball games a year together, back in my hometown. I now live about 3 hours away, and often find out about friends attending the same game on Facebook. She'd much rather stay at our seats watching the game than wander the stadium with me to meet up with people who for her are acquaintances at best.
I think, maybe, if you're with someone who isn't into baseball enough to be entertained by the game itself, it would be rude to leave them alone for long (and in that case they might be more willing to come along and meet strangers), but a baseball game can be a long event. I know that we often get to to the stadium 1-2 hours before the game to watch batting practice and warmups, once you add the time of the game to that you can be at the stadium for about 5 hours. Stepping away for 15-30 minutes to meet with some friends doesn't seem excessive.