I had the same when I accompanied my younger brother on a nursery school trip.
He was 3 and I was 16, and I noticed fairly early on that none of the other mums were talking to me, but looking at me a lot - then talking quietly amongst themselves.
It wasn't until we all sat down for lunch, 3 hours later, that one of them finally asked me if I was the mother. When I said no, I was the sister, I was suddenly included in all the conversations and they didn't go quiet whenever I walked near to them. I had spent 3 hours being ostracised by a group of 15 older women who had made an assumption- and it was really intimidating!
It didn't occur to me until later that mum had never dropped my brother off - but that it was always my dad (45) that did, and so the women must have known him. What they must have thought about that... well, I don't like to think about it!
On the plus side, my mum - who was 41 when she had my brother - has never had any remarks about her being the grandmother, only ever the mother. Which is nice