Frankly, if this is the norm for the OP, her friend and the kids, I have to say that the 12 year old is plenty old enough to deal. She's not a stranger to him. Sounds like they all knew this was the plan from the outset. This was nothing out of the ordinary. For him to come back within 10 minutes, come in the house, mope and then cry seems like a reaction that's way out of proportion, given the "normalness" of the situation and the age of the child.
Maybe he doesn't get along with other kids. Fine. Take a book, take a walk, sit under a tree and count the leaves. But when the adult that's in charge of you for the day tells you to do something--once again, something normal that these children have done many times--you do it. The kid is 12 years old!
The OP did all of the right stuff when the kid came back. She tried to ascertain whether he was hurt, whether people were being mean to him, etc. He barely responded. In my book, if you don't do what I tell you, will barely communicate with me, and then when that bare-bones level of communication elicits tears for seemingly no reason than not wanting to follow directions, that's just a bit too much.
OP, I'd say that the kid is either old enough to stay by himself next time or his mom needs to tell him who's in charge and who sets the rules. If he doesn't like what you tell him to do, then he needs to deal with it or communicate with you and come up with an alternative. Like I tell my kids, if you want me to treat you as if you're more mature, then act like it.