It's an unfortunate situation, but for the reality check, it's not your situation. It is totally okay that you don't want to be responsible for these kids, even for a few hours. Even if they were older and perfect angels, it would be totally okay. *I* don't want to be responsible for anyone's children, though I personally would make an exception for a true emergency, and it would really have to be an emergency for anyone to call *me*!
So, I think you have to decide what your boundaries are (with your DH, too). Can you tell friend you're okay watching the kids in an emergency? What about as a last minute backup? For literally just two hours once a week? I'm not saying you have to agree to any of these, I think you can have a zero policy if you want, you just have to decide what you're comfortable with. Then next time Friend asks, maybe you'll be able to tell him, "While I'm not comfortable watching the kids on a regular basis, I am happy to be a last minute backup if you need me," or whatever is applicable.
A zero policy is a little trickier, I grant you. If you suspect Friend is looking at you for help in this regularly, you might be proactive and say, "So, what are you going to do about getting people to watch the kids?" just as a matter of friendly interest. You could recommend babysitters or daycare providers to him, for example. And if he says, "I was hoping you would watch them," you could say, "No, I'm sorry, I can't do that." I think I really would stick with "that won't be possible," even though it sometimes seems more natural with friends to explain things. I think your reasons are perfectly valid, but it's not the sort of thing a newly-single parent is going to find helpful, you know? So I would try to avoid explaining why, even as you stick to your boundaries.