I get the sense, and I might be totally wrong and feel free to tell me if I am, but that you might feel a bit guilty about saying no to your church family.
As you point out this is how you give back. You can't, for very understandable reasons, do what others can do. Be because of time or space, it happens we all have things that we'd love to do for others but for a variety reasons just aren't' able too. So we find ways in which we can give back.
It sounds a bit to me like you feel guilty or bad about saying no to, essentially, giving back. I don't think you should feel guilty. Giving back doesn't mean always giving at the sacrifice of yourself. You need to go to these events to reconnect with other adults, to feed your spiritual self, and to have a good time. And giving to yourself, is just as important as giving to others.
I also second the others who say, they mean no harm. When they ask, let them know you'll be attending the event as well so won't be available to babysit. I'd put down money their reaction won't be "Oh, but, you're single so we assumed you'd take the kids", but it will be "Oh great! We can't wait to see there!", at most they might ask if you know of anyone who'd be willing to babysit.