I've never had anyone quite this persistent, but my way of brushing off people asking for an invitation is usually to say something like, "Oh, it's just going to be me and [person], actually. Sorry." No justifying it, no explaining, and do not add the phrase "this time" if you don't want them to think they might be invited next time. If they're a person I genuinely do want to spend time with, then I offer to schedule something with them another time. If not, I just leave it at that. I've been in some social circles where inviting oneself along to things was very common, and that response seems to work well without seriously hurting anyone's feelings. The key is to keep your tone sort of brisk and polite and then move on from the subject quickly. Don't act overly apologetic or wait around for them to respond - you want to make it clear that you expect them to take it well and be reasonable about it, and you're so sure they're going to react that way that you don't feel the need to check for their actual response. Most people pick up the cue when you do that and follow your lead onto a different topic of discussion.
If I get a "but whyyyyyyy" type answer or they try to argue about why they should come, I just act a bit uncomfortable, and then say, "That's just how it is" or "No, that won't work" or just change the subject without responding. With the few very obnoxious people I've known, I just tell them no once and then never engage on the topic again if I can help it - whenever they bring it up, I just give them a slightly confused look and change the subject.
If your friend's friend is really so persistent that she can't beandip her way out of it, she can always try saying, "I'm sorry, but this is very awkward for me. Would you rather wait to talk until after I get back from my vacation?" And then when friend inevitably tries to argue her case again, just say, "I already told you it won't be possible. I'm sure you can find your own vacation for the same timeframe. Let's talk again after we both get back." And then she needs to leave, and not engage on the topic again unless it's to say (with increasing impatience), "We've already talked about this." Basically, once she gives the initial "no," she should just refer back to that - this friend needs no new information, because she already has all the info about the trip that she needs (namely, that she is not invited).
Personally I think it's OK to say "I'm sorry" if that makes your friend feel better, but she shouldn't linger on the apology or be overly conciliatory about it - just say the words, since they're so ingrained, and then
find something else to do. I find that's the best thing for people who have trouble saying no - find lots of things they suddenly need to be busy doing so they don't have to deal with the aftermath and can get their nerves under control in private.
Really? I know this woman was clearly rude, but if my friend has been telling me about her great beach vacations for years & I say to her one day "you know, I'd love to join you guys sometime, if you're looking for new people to join in. But if not, no worries.' Would that be rude?
I know that's not a question, per se (and I gave her an out) so does that make it different/less rude?
I would never randomly ask/demand someone that I barely knew (like what happened to the OP), but if I was close enough to someone that they were sharing details of their trips (or of their life) with me, then it wouldn't strike me as rude to make a question/statement mentioning that I'd be interested in X thing too. Am I incorrect in that thinking?
I think that's fine, though it might be better to let her give you an opening before asking. If someone says, "We usually vacation with So-and-So, but they can't come this year," then it makes more sense to say, "Oh, well, we're available if you're looking for someone to help split costs." But I do think your way is fine since you didn't say "Can I go this year?" but rather, "If at some point in some hypothetical future you're thinking of inviting someone new along, we would be interested," which is rather a different thing IMO.