I think it'd be okay for ONE additional person to join someone at the front of the line, but two or more should do as you did and just go to the back of the line.
A similar thing happened to my mom and me once when we were in line to place our order for lunch. There was more than one line and we picked the one we did based on how many people were already in it. I wasn't too upset when a woman joined the man in line in front of us, but was VERY perturbed when their TWO friends joined them. I think I made a comment about where the end of the line was, but the man in the Entitled Couple (the ones who cut in line) was SO RUDE. He made some comment about his time being more valuable because he had to get back to work (he was in dress pants and a button-down shirt and my mom and I were dressed casually). And I responded, "Oh, my bad. I'm here because I evacuated New Orleans because of the hurricane." (It was a year before Katrina -- Hurricane Ivan that time and I was in Houston at my mom's house because of it.) The jerk's wife became very apologetic, but, of course, not sorry enough to go to the back of the line. Anyway, I was super peeved and applaud your good sense not to cut in line.