Thank all of you so much for your replies. I have read each one here and I appreciate each one of you for taking the time to respond. You have no idea how distressed the BIL has made me over the past 5 years.
Sharnita, yes, DH and the BIL's parents are deceased.
There are many people who commented who I want to respond to. All of you, really, and you all said things that have already helped me think in new ways and I'm reading going: "Yeah! That's right! They understand!"
I agree with those who said DH is just as responsible. Yes, he is a nice man, but all of you are right, he should have fixed this long ago and straightened it out with his brother.
DH always wants me with him in the presence of BIL because he says he thinks that otherwise, BIL will think I am "scared" of him (BIL). I assured my husband that I am most definitely not afraid of his brother, I have just reached critical mass in how much more of his creepy (what a fitting word to describe it, someone used it here -- perfect description! Ha!) I think DH never had the courage to stand up to the BIL before and he thinks that together we will do that.
However, on the other hand, I completely agree with those who say that no matter what I say to the BIL, it won't change his behavior one bit. I agree with the person who said he's obviously been at this sneaky abusive behavior for a long time, and is an expert at not getting caught.
I also agree with the comment that the only thing BIL'd really fear would be being exposed to his circle as being what he truly is - an abusive, deceitful bully. He wants everyone to see him as such a good 'Joe Citizen' type. If they only knew.
DH and I had a long talk this afternoon about how to proceed. You all have no idea how much I want to put BIL in his place. But this man is so entrenched in the art of denial. He says, "What did IIIII do?" and makes a face like the person accusing him is crazy. Some of you here have already made mention that bullies/abusers do that. Oh yes! This man is the same. So- DH and I decided it's hardly worth a confrontation because - as yet another poster explained - BIL might "get off on it". And the BIL does seem to take great delight every time he thinks he has intimidated or upset me. It's like he strives for that.
I agree also with the poster who said keep anything said to this man as short and terse as possible. "Yes." "No." There is no reasoning with him - he will never accept responsibility - why waste your breath?
So DH and I have begun the distancing procedure. We are both in agreement with it. It actually involved me having a birthday dinner for the BIL next month, which, according to DH is the very last time I do this. The same for the Christmas meal (one of them) as well.
By the way, what really cinched this and goaded me to finally post today is, BIL misled DH and I and played us against each other, that he had been invited to our house for Christmas.
I am meticulous about my invitations. So I wrote BIL and said: "Things have been so hectic here lately, I can't even remember sending you an invitation. Can you please tell me if dinner was at 4:00 p.m.? --- we look forward to seeing you at 4:00 p.m.!" --- he wrote back, and completely ignored my question. This clued DH and I in to the fact that something was very dodgy. DH spoke with him, and asked if he had been sent a Christmas invitation by me and a menu. BIL said no! He said: "I just thought I was supposed to come over there for a meal." DH said: "Who told you that?" and BIL said: "I thought we were going out to eat at one place before Christmas then I was coming around to your house." BIL never committed himself to who invited him.
Of course I was livid. He had misled us and yet I had already welcomed him (stupidly!) in good faith to come for dinner since he alluded to the fact that he had been invited. What happened was, we were all discussing possible alternatives to eating our on Christmas. He (said) he took that to mean he was invited to our house. He wasn't.
So, the blatant untruth did it. DH wants me to cook BIL's birthday dinner and a Christmas dinner for the 3 of us - then next year, there will be no birthday dinner and no Christmas dinner. He has promised to talk to him less on the phone (he gave me specifics) and see him less regularly (specifics on that, too).
All of you have helped me so much. What a wonderful moral support, because really, I was so miserable over all this and at my wit's end. I don't really have anyone to talk with about this.
I really appreciate every one of you for replying. I will be updating about how things go, and later on I will read through other posts and if i think I can help in some way, I will respond.
All of you really gave me great support when I needed it most. I mean it, thank you every single one.
P.S. To those who said just let the husband see BIL any time he wants and just don't go yourself, I am in complete agreement! But DH often insists I be there. He has this thing about not wanting BIL to think I am afraid of him. I have really pushed the issue that they can go out anytime together wherever/whenever - but DH has said: "I don't want to be with him!" Isn't that peculiar? But yes - I would totally sanction them going on vacation together - but it's clear DH doesn't enjoy his company much either.