My ex and I began to have less and less to do with some friends of hers, after an incident with our youngest son.
We were at a restaurant with this crew and both our kids (younger one, Zack, was only a few months old at best.) I had to take the older one (Grant, about 18-19mo at that point) out of the restaurant because he was getting fussy. Shortly thereafter, my ex comes storming out of the restaurant, holding Zack and hysterical. It turns out that one friend was holding Zack when another asked to hold him. First friend passes him off, but the second one missed the catch, dropping Z a few inches to the table. We rushed off to the hospital -- fortunately Zack was fine. We'd had some issues with these people before, inviting us to a New Year's party and not warning us that the hosts had dozens of tiny fragile glass knick-knacks all over the place. We ran ourselves ragged trying to keep Grant from breaking things. Plus, people would say that they'd watch him for us, and then just "forget" to actually do it.
Bottom line: Safety and your peace of mind trump just about anything else. If you don't trust people to hold your child, then don't let them. If they get their feelings hurt, that's their problem, not yours.