If you're doing written invitations (recommended), put 9:30-12:30, or even 11:30, if you want a chance to "cool down" before you have to go to something else. ETA: works for Email too, as long as everyone actually checks their Email.
On party day, you and your DH subtly start cleaning up, picking up trash, putting things up. If that doesn't work, there's nothing piggish in my opinion in saying, "Gee, we are so glad everyone came, but we have something going on this afternoon. We'll have to get together real soon!" Given it is a holiday, the other guests might be busy, too.