It might be time to come up with a standard answer. Either something calming like "I just don't, people are different, but go ahead and enjoy yourself," a factual "I already told you why" and change of subject, or actually ask him "Why does it bother you so much that you're more into this than I am?"
You've tried explaining yourself, and it doesn't seem to be getting through, so maybe it's time to turn it around and get him to explain his position. Not as "how can a grown-up care so much about this?" That would just make him defensive, and it wouldn't stop him hassling you. The issue isn't that he cares more about Halloween: the issue is that he seems to need you to care about it as much as he does.