Ok, so I think that there are *several* issues here.
The first one is the simplest: communication. OP, I totally understand where you are coming from about not picking up on the cues of others. I'm the same way myself. One thing I now do with my spouse is to simply ask what they mean by what they say.
So, "I'm tired" could be met with, "do you want a cuddle, or do you just want me to leave you be?"
Active listening is another good technique. Repeat the gist of your partner's words back to them, and see if it's right. Of course, you don't have to agree with him, but at least you will understand what your partner's position is. It's great for opening up a dialogue.
e.g. "I feel really neglected."
"So, are you saying that you want me to spend more time with you?"
The second one is the problem of him leaving the toddler with you. My solution to that is also quite simple. Every time he does this, he owes you 'one'. Have a chart where you can mark each time he owes you 'one'. Once he gets to a pre-determined number of 'ones', he can't have any more until he's paid some back, by giving you the equivalent amount of toddler-free time.
The last one is the hardest, and something that my partner and I struggled with: you both have a right to your feelings, and you don't get to be annoyed that someone else is annoyed. People can't turn their reactions off just because you explain why you are acting a certain way. "Sorry I smacked you in the face sweetheart, I didn't see you when I was turning round." Well, that explains what you did, but it doesn't stop the other person's nose from bleeding.