I don't really think there is a statue of limitations on when "past wrongs" can be dealt with. Obviously, it makes more sense (at least to me anyway) to deal with things as they come up, but not everyone has the courage or the ability to do that.
I get that you felt that he had attacked you and you weren't expecting it. But to give him some credit, he did have a private conversation with you, away from others. He didn't berate you in front of the rest of the party. I would assume that because he is moving away, it is easier for him to confront you, because he doesn't have to interact with you again if things didn't go well. I also imagine the question "why did you say those things?" is so that he can understand what was going on and perhaps improve his relationships at his new location.
So basically, I don't think it's necessarily rude to bring up something that was hurtful just because time has passed. I think this can be done in a rude way, but that's regardless of how much time has elapsed. If I were you, I'd take it as a learning moment. You didn't realize that you had done things to make this guy feel bad and now you know. It's a good check for when you interact with others in the future. (Now, if he had confronted you about these before and then brought them up again, I think it would be very different).
And all that being said, I don't blame you for how you feel. Showing up to say good-bye to someone and having them tell you, suddenly, that you've been hurtful to them in the past is unexpected.