I would say 2-3 times also. I'm thinking of someone I don't know very well, that I'd like to get to know better. If they decline 2-3 of my invitations without making any effort on their part to move the friendship closer, I would take that as a sign they weren't interested. That's not to say I would give them the cut direct after that or anything, I would just leave the ball in their court, saying something like, "Ah, too bad. Well, let me know if you want to get together some other time."
Actually, I think for me it would be generally the same with long-term friends as well. 2-3 times, and then I would be like, "Hey, life seems crazy right now, let me know when you get a breather so we can catch up!" I wouldn't want to feel like maybe I was pestering them, or reminding them of stuff they just don't have time for right now, or whatever.
I think if you're wondering if you should try again, a good test might be--don't. If you normally invite them to something once a month, resolve to go three months without doing so, and see what happens. Do they contact you? Do you miss them? Then you can decide whether to start inviting them again.
I think sometimes with family it can be different, because sometimes one is obligated to invite "all family in the area" or "all family at a certain level" to things. For example, I haven't seen my wife's cousin in years, because she simply won't come to our extended family events. I don't know why. My cousin and their kids come; they don't live that far away. If I were hosting an extended family event, I would certainly still include her in the invitation, even if I would be shocked if she showed up. But at the same time, in that situation I wouldn't be keeping count of her declines or stressing about it, as long as I got her decline by the deadline (if applicable).
It might be similar with, say, all members of a book club--you always invite everyone to your "post book club supper" but there are a few people who always decline. If I was inviting everyone because they were in the book club, I would keep inviting those people, but just would stop being invested in whether they accepted or not.