Re: the party that just past, personally I wouldn't draw attention to it. If someone should happen to mention it to you, you could say, "Yeah, I kind of botched that, sorry," but I wouldn't preemptively apologize to people. Some people probably didn't really mind and others might have thought the parents just rudely showed up with their kids without prior approval, and even though it's okay etiquette-wise to turn someone away at the door, people understand that's hard to do in real life, so they might have thought it was that kind of situation. If anything it makes the ones who brought kids look ruder than they really were, since you told them it was okay, but again I think it's pretty awkward to bring it up--if someone says, "I can't believe Sally brought her kids to your cocktail party!" you could say, "Er, actually she asked first and I told her it was okay. I know, I shouldn't have done that..."
I'm no party-person, but I've been to a lot of casual gatherings that were family-friendly but not specifically geared towards kids (the way a child's birthday party might be). Perhaps you could start by listing what, for you, makes a party adults-only vs. child-geared, and use those lists to meet in the middle. For example, I wouldn't rent any kid-centric equipment (like bouncy castles), but I would get the box of sports equipment out of the garage and maybe pick up some dollar store bubble solution. Kids will figure something out when they're playing outside. It would be good if they had a place to play, like the backyard or the basement, where they were somewhat separate from the rest of the party (so the adults could chat) but also easily accessible to their parents.
On the other end, I wouldn't have expensive, hard-to-eat food like whole lobster--I would go with hamburgers, tacos, pasta, etc. that can be easily eaten by someone without the best dexterity. You can still have some pretty sophisticated food that way, but I would just make it more low-maintenance. I would set it up buffet-style and let people sit where they wanted--place cards, courses, this wine must be drunk with this food, etc. would be too formal for a casual family-friendly event, I think, and too easy for kids to interrupt, thus irritating people who would wonder why the kids were there if this was the party you wanted.