Why can't you simply say, "I'm going to have to skip the gift exchange; I can't afford it"?
Frankly, I'm not sure I'd assume that the gift exchange is mandatory, even if nothing was specifically said about it. Don't buy a gift, and don't take one--that's really the only important thing. Say something quietly to the person putting the names in the hat (you don't even have to give a reason)--"I'm not going to do the gift exchange, so be sure to leave my name out."
If there's no name-in-the-hat, then just don't bring one, and when it comes time, say quietly, "Oh, I didn't do the gift exchange," and position yourself to not take a gift.