I think it's totally up to the tour guide to make sure that the people on his/her tour belong. At the beginning of the tour, the guide should take physical or mental inventory of the group and keep an eye on that for the duration. If people who are not part of the group spontaneously join the tour, the guide should politely inform them that it's a private tour.
Or, possibly, if your manager knew that these folks weren't part of his group, he could have taken the tour guide aside and descretely asked if they were supposed to be included in the tour.
Barring that, I'd say, there wasn't much you could do but grin and bear it.
To answer your question, though, if they didn't pay for the tour, then yes, they were rude to join in like that. Even if they thought it was just a regular museum thing for all customers, they should have at least asked before insinuating themselves into the group.