Don't go, citing lack of food you can eat, or offer to bring your own meal, since they are not serving anything you can eat. I usually end up doing the latter. Sometimes I do it in the form of something everyone can share, like, oh, Mac and cheese or baked beans or gyoza or something. Sometimes, I bring my own bento/tiffin box just for me, which sounds like the better deal in your case. I don't do it without first declining and saying why, and, if pressed, offering to bring my own.
"I'm sorry I won't be able to come. I'd enjoy the company, but you aren't serving anything I can eat, and I certainly don't expect you to alter your menu." "Oh, but...." "Well, I could bring (my own/complimentary dish) if that would be OK with you." It is usually a very simple conversation, and I very rarely stay home.