I'm having a cognitive disconnect here. How can your absence from an anniversary memorial service for a child possibly ruin the "fun"? I've been to memorial gatherings that *were* fun -- lots of sharing happy memories of the deceased -- but I can't imagine such a thing when the deceased is an infant.
Growing up Catholic, I went to my share of anniversary memorial Masses, and they were pretty generic: The priest would say, "We pray for the souls of Your servant Martha and all the faithful departed" instead of the generic "We pray for the souls of all the faithful departed." That's the closest thing to a death anniversary service I've ever attended -- unless you count Good Friday services.
Grief is very personal, and if this child's parents want to honor her memory with an annual service, that's their right. But they have to understand that not everyone will be able or willing to attend.