I'll weigh in with the childfree opinion. No, looking isn't rude. I don't know why, but small children tend to stare at me (nothing unusual about me). No biggie to me, looking isn't invading my space. If they are grabbing at me, yelling at me or pestering me in another way, then I got a problem with the kid.
But quietly looking? I can ignore that easily.