I think I would just plan to have dinner there as a family, with no guests. If someone happens to say, "What are you guys doing for dinner tonight? Anything special?" you can certainly tell them, it's not a secret. And if they say they happen to be going there, too, you could say, "We're going right at five. If we arrive at the same time, do you want share a table/get tables next to each other?" But if it gets any more organized than that, you would have to make it clear you weren't paying for it and it's not an extension of the party so much as a happy coincidence. It might seem a little A-list/B-list to other people if it looked like you had planned for a subset of guests to go on and do more party stuff while the others went home.