I'm not a terribly assertive person in general, far from it, but I feel so exhausted and cranky if I don't eat that I have no problems telling my friends, "Dude, I'm starving, I'm going to get something to eat before I keel over." Maybe not in those exact words, but that's the gist. Even if we're just technically hanging out and I wasn't specifically invited over "for lunch" the way the OP was.
Expressing hunger isn't something I feel all that shy about, lol.