I admit, I am usually the one who is trying to organize these groups, however I have never judged anyone for their inability/unwillingness to join in. If you want to go, then you should join them. If not, then its not a big deal. If you can afford it, and have no ethical/moral/etc reasons not to, I would donate at least a small amount to the charity. Even 5 - 10$ will make it seem like you want to join in and smooth over any hard feelings.