If it makes you feel better, DH and I have maintained that our marriage comes first, above all else, even our daughter whom we struggled very hard and very long to have. Most people in our family are aghast at the fact that we take an annual vacation without her because having her was such an uphill battle. Very few seem to grasp that the very best gift we can give her is a loving and lasting marriage.
Regardless of whether you have children or not, your first priority is your husband and marriage. Honestly, I'd have a talk with the dissenting parties and let them know, now, that either the complaining will stop or you will spend your holidays at your home, alone. Once you make and follow through on that ultimatum, you may have one or two holidays you enjoy alone, but usually the complainers/whiners fall into line.
Even though all of our family is in reasonable driving time of each other (less than an hour) we still split the holidays and change them every year. Christmas with one set of family, Thanksgiving with the other and the next year we flip. My Dad has recently decided that he does not like my in-laws and will not spend time with them--thus he spends most holidays alone because even though I don't like them either and tensions between us and them are strained, we still have them over. It's too bad if my Dad can't deal with them once or twice a year.