I wouldn't worry about it until after the RSVP date. Personally, I'd cancel it if no one RSVPs by then. But you might want to call everyone to warn them that you cancelled it or I'm afraid you might get someone on your doorstep expecting a party. Or you could just call everyone and ask if they're coming.
But see, that's just feeding the rudeness beast. I think that calling/emailing people and begging for a response is encouraging rudeness. I also think it's a terrible idea to contact them and tell them the party is cancelled. No RSVP = they are not planning to attend, so why notify them that a party they don't plan to attend is cancelled? Why would they care?
It's entirely their own fault if they show up for the party only to find no one home. Perhaps wasting their time and gas will be a motivator to actually respond to invitations!
In some ways I agree with this--why SHOULD you have to chase people down who couldn't respond in a timely manner? But on the other hand, it really depends on the social circle. In some circles/families, and for some events, no response = not planning to attend. But in others, no response = YES planning to attend--like, you should know we were of course planning to attend! And actually in my family it could go either way.
Which I'm not saying is
polite, but them being rude leads to practical problems. Yes, it's perfectly fine to turn unexpected guests away at the door. But I personally think that's one of those perfectly fine things that is really, really hard to do in real life, especially when it's a close relative trying to attend a child's birthday party, when they can see other relatives celebrating that birthday just inside the house.
What could work, though, is the host deciding to change the location of the party based on the 8 guests of 40 who have responded--not to a bait and switch extent, but maybe going to a restaurant instead of feeding them at home.
Then the non-respondent who just shows up will find themselves facing an empty house, and no idea where to go next. And I guess the host can ignore them if they call their cell phone (though other guests might not and the person might find you anyway). If the host decides to cancel the party entirely, it might be helpful to leave the house during the party time as well, if they fear non-respondents showing up at the door.
But honestly, this kind of thing happens all the time in my family, and no one ever learns from it.

The house is dark, you have to call four people to find out what restaurant they went to, you get to the restaurant and have to drag up an extra table so you have a place to sit, people say, "Oh, I didn't realize you were coming!" and people STILL don't get a clue and do it again next time. Fortunately my parents and I are not usually in a position to host these things so it's not our place to correct the offenders, but I imagine it's aggravating to the hosts. (But not aggravating enough to prevent THEM from doing it when THEY are guests. Oh family. Fortunately they have many other redeeming qualities.)