I agree that it can't hurt to say why you aren't coming. As a frequent host, I like it better when someone I've invited to something tells me why they are declining. Of course, they're not obligated to, but it just feels, I dunno, more genuine to me.
Sorry can't make it!
as opposed to
Sorry can't make it because we'll be out of the country that weekend.
The second one just sounds more regretful and sincere. Totally just my own perception on that, I should add, but anyway, how can it hurt?