My opinion is that you have the right, as the host, to make your own guest list. (That is, you have the right to invite or not invite particular people into your home.) So, yes, you can make the decision to not invite them.
However, as others have said, if you make that decision, there may be some unpleasantness from others who disagree with that opinion. I guess you'll have to decide if the unpleasantness of having them in your home is worth it to avoid the unpleasantness that may happen if you exclude them.
Unless, as someone else suggested, if you haven't set the date yet, would it be possible to schedule the party for a date when the unpleasant relatives are not in your country? That would be an easy way to avoid trouble.