There are two separate issues here, really.
One, inviting someone who isn't a member of the family along on a family trip.
Two, renting a cabin and then inviting more people than the cabin will hold.
For the first, I think it's okay for your mom to invite a close friend of a member of the family along. I'm very introverted too, but as long as there was an escape route for me, like a room I could retreat to, or being able to go off by myself for a few hours every day, I'd be fine with that.
For the second, I don't know why people invite more people than there are beds. Unless the person extending the invitation is willing to take the floor/air mattress/sofa. It's just plain silly to rent a cabin that clearly sleeps 4 people and then invite 5 (or more) people to stay there. This is the issue I think you should address, because if you can solve the problem of the sleeping quarters, it will be much, much easier to deal with the boyfriend's presence.
Keeping my fingers crossed that your dad can find a larger cabin.