This is one of those times where, without mincing words, it's fine to be direct but aiming for peace.
I have a couple of friends that I've bluntly told them that their child can't play with mine for target practice for guns or bow.
We have shotguns, a rifle, a muzzleloader, a crossbow and a compound bow. All used for hunting. We take the owning, storing and using of these weapons seriously and have drilled it into our kids' heads as well as had our entire family take a safety course.
So why can't DS play with little Jimmy (this was when they were 13 or 14)? I saw little Jimmy pick up his rifle and wave it around, including in my direction and when his uncle noticed me getting ready to dive to the ground, he absently told his nephew to quit waving it around. I yelled at Jimmy that he should never, ever, ever, ever point a gun at someone unless he actually planned on shooting them. Jimmy said, "It's not loaded," to which I countered, "There is no such thing as an unloaded gun." Based on that alone, we were NEVER going to allow this kid to be anywhere near us with a weapon.
Then DS told me on the way home that Jimmy accidentally shot him in the throat with his pellet gun.
So I had to call up the whole family (this bunch is like a 50-headed hydra) and explain that since all the adults neglected to properly instruct Jimmy in gun safety, he was never allowed to be anywhere near us with a weapon. The kid just got his own crossbow (shudder). They didn't like it and I didn't care and our kids are not close. Oh well, safety trumps everything.
So OP could say to neighbor, "I'd like to give the girls an opportunity to spend time together at your house but you do X*, which our family is not comfortable with so I'd prefer they spend time together at my house, school or the park. I'm not saying you shouldn't do X, just to be clear."
I'd be a little startled to be told, for example, because we have weapons, a child was not allowed at my house, but NOT offended. Most everyone around these parts has weapons and I'm not the only female here that hunts.
* Even I would have a hard time saying 'my DD can't come because you are mentally unstable'. But if there is a practice going on at their house you are not comfortable with, it's best to focus on that.