Freaky, asking if you minded if he joined you is not persisting. It's asking your permission to join you and your family at an activity. Your choice at his question was simple - "yes, Pedro, please come join us" or "no, Pedro, this isn't something you can join us at." You chose the former. You may not like that he took you up on that. But, you can't be angry at him for coming when you clearly said he could. It's your fault alone.
I know you're upset that your family time was taken by someone outside the family. But, the only person you can really blame for that is yourself. You chose to allow him to come. You, and you alone, are the one who said "yes" instead of "no."
Pedro was not rude in anything he did. You want him to be so badly. But, he wasn't, Freaky. Nothing he did was rude.
And, you weren't rude, either. But, you were misleading. And, non-communicative about your true feelings on the subject. But, that wasn't Pedro's fault, either.