I think the missing pieces of the puzzle are
1. What is the relationship between the two? Are we sure that this was a father talking to his son, or could it have been an older brother, cousin, neighbor, babysitter? Now, I'm not saying that a different relationship would automatically make it right, but I do know that I can say things to my siblings that my mom can't get away with saying.
2. What tone of voice did he use? I'm guessing that we never would've heard this story had the man in question not been angry. But, I think a lot of us are reading it in the tone that we would actually say it (if we would) as opposed to how it was said. I know if I tell my brother he's being a pain, he knows I'm annoyed with him but there's no punishment coming. It's just time for him to go home. And I would never use the whole phrase. If there is only one place in the entire world I manage to watch my language, it's in front of my siblings.