I think you are being oversensitive about his comment. And I think you have been making a mistake in pretending that these people are like family to you. You need to back off and not consider that they are people who are going to be sensitive to you. They are going to be who they always have been and you are not used to it!
In other words, part of the problem for you is that your expectation is that they love you and will act in a loving manner. Well, maybe they love you in their own way, but its not a way that works for you.
For about the first 15 years of my marriage, I tried to consider that my MIL and FIL were really wonderful people who really did care for me. They would write that they loved me in cards and it always made me feel really weird when I read that because its not how I felt. I felt the whole thing was forced.
Add to that that my FIL refused to talk directly to me for about the first 10 years and that my MIL had a number of sisters who compounded the situation and I was not only outnumbered, but my DH didn't want to deal with it.
After 15 years I gave up. I didn't care what they thought of me! Finally after 20 years, DH had the same epiphany.
So, when we visit, I don't treat them like family. I am reserved and polite. And what's really really weird is that they practically kiss my butt! I don't get it!
But, I can tell you that my problem with them (for me) went away when I stopped caring what came out of their mouths!
I was reserved and polite. I bean dipped, avoided problematic subjects, backpedaled like mad and even excused myself upon hitting a hidden land mine.
For all those years I was also raising children and we would talk to them about their grandparents. Yes, they love you and you can love them, but they are kind of nuts, so keep that in mind. They did. One of my sons is finally getting to know them a little better now that he is an adult and living near them.
But, still, he avoids certain topics. He tries to find the commonalities he has with them so that he can have some kind of relationship with them.
My youngest son was always a target of sorts for them. Middle son was the favorite (still is--the one who lives out there now). And there were times when FIL was a jerk to my youngest son and I got in between the two of them like the great wall of China! Boom! You don't mess with my kid!
Youngest son is now an adult too and participated on a vacation activity with the grandparents and his cousins (the only other and much younger grandkids). One of those cousins, about 10 years old picked up on it and asked her mom why her grandfather didn't like her cousin (youngest son)!
But, to youngest son, it just rolled off his back. He didn't care anymore. It doesn't hurt him. He takes what he wants from the relationship, is cordial and really doesn't let it bother him at all.
The big losers? The grandparents, of course!
My father did everything he could to establish and maintain a relationship with my kids. He treated my husband better, I think, than he ever treated me! My DH could do no wrong in my dad's eyes. My mom's too.
People make their own happiness and they have to be responsible for forging their own relationships--real relationships that have give and take, that have listening ears and open hearts.
Don't pretend that's what you have when you don't. They don't have to love you--your DH loves you--he chose you! So, who cares what FIL thinks! He is the one who is missing out!
You can't have a relationship with someone whose road has a giant One Way Only sign on it.
So, visit away! And act like an unrelated guest. Be cordial and even somewhat distant. And protect yourself.
My advice to him would be for him to open up and really listen and to actually forge a real relationship with you. But, since I can't make him do that, you have to protect yourself.
Love your husband and respect his parents for raising him, but don't go any farther than that. I understand that you don't want to make it harder for him with his heart thing and whatnot, but he would not give you the same consideration--MIL didn't when you had a miscarriage.
So, whether you drop them in reality and don't visit them, drop them in your heart. It's not worth it. Loving people do not act like that.
Again. Protect yourself.