Having been in a very similar situation, I POD those that say tell your kids no. If you already know Grandpa does not respect boundaries, then don't put yourself in the position of being the "bad guy" when you have to enfoce those boundaries. Be prepared for the fall out, it likely wont be pretty, but just remember.. You're the mum. You get to make the rules for your kids., No-one else has to agree or even like them, but if they want unsupervised time with your kidws, they do need to respecct the rules.
Why not ask Grandpa if he would like to stay at our house for the night?