If it makes you feel any better, I had almost the exact same situation with my daughter's birthday.
RSVPs were due the Monday before the party, which was on a Saturday. The people that I expected to be there did RSVP by the deadline, but I never heard from several people, who didn't show up. One little girl's mom told me the day of the party that they were coming - about an hour before it was supposed to start! The girls play softball together, why the mom didn't tell me on Tuesday at practice, I don't know. She told me as the game was starting on Saturday morning that they were going to be there, but possibly a little late due to going home and changing after the game. Luckily, the party was held at a paint-your-own-pottery place, so it wasn't any extra work to add an additional child, as the pottery was the party favor.
Since we only do "friends" parties every other year or so, I can deal with the flakiness, but it's really annoying trying to coordinate everything and then finding out at the last minute that you're going to have extra people. Happily there was enough cake and ice cream to go around - plus left overs.
