I'm pretty much the same as you, lowspark. I don't do gifts except for a couple of people (my best friend and I have a tradition of exchanging small, fun items and as we collect them year round it's not difficult!)
Here's the thing for me, though. If I tell someone 'I don't do gifts' and then they get me something because *they really want to*, I will accept it graciously and enjoy it. Especially if it's food! So long as that person truly doesn't expect reciprocity and isn't going to feel slighted or secretly bitter that their giving me a gift didn't make me break my 'no gifts' policy for them.
The reason I don't do gifts is that I find it very stressful, and often in the past, when I've picked out a gift for someone I've thought was pretty cool, I've seen that look of disappointment in their eyes, or accidentally hit on an issue I didn't even know was there. I'm just no good at finding things that work.
So if I were this person I would be completely fine with receiving something from you in the spirit with which it was intended! But, that's me, and obviously I can't speak for all the no-gift people.