I think it's one of life's little awkward situations, where even if you had multi-angle video footage and some diagrams, there wouldn't necessarily be any clearcut answers.
You really just have to go with your gut, I think. Nobody likes waiting in line, and I think in general it's rude to hold up the line. As the drifter moves up in line and gets closer to being "at bat," I think the onus is on them to pay attention and be ready to start with the cashier promptly. If the drifter isn't in position, and the next person is standing there waiting, I think the next person ought to just go ahead, and the drifter can't complain about it. Maybe the next person will let the drifter go in front of them, or maybe the drifter has to go to the end of the line--you snooze, you lose kind of thing.
It gets trickier if we're talking about positions 6 and 7 out of 10, though. If I were #7, I think I would always let #6 back in front of me (if they wanted), even if they'd drifted completely out of view, as long as we weren't at the point of setting things on the counter in front of the cashier. But some people wouldn't, and I don't think they'd be completely wrong. I think if your place in line is really important to you--like you need to get home by a certain time--the onus is on you to stay fully in line, and not introduce any ambiguity into it.