I don't understand why you feel you can't speak to the boy in question directly. At the time it is happening a quick "I'm praying right now, we can shake hands later" would've been perfectly fine and he would have been able to understand what was happening. Just because he has a disability doesn't make him incapable of understanding an adult. You said he was a teenager, not a small child, talk to him like you would any other teenager.
I'm guessing that the OP wasn't comfortable speaking out loud during a time when others were engaged in a quiet moment of prayer, so tried to use non-verbal communication instead.
Very true - I did not wish to speak up during this time and had to use a form of non verbal communication instead. For many Catholics, they prefer the time just before they receive and after receiving Communion to be some form of quiet time for them to reflect and/or pray. I did not feel that I should have said something to the young man at the moment as to not to bother others who may be spending time in prayer or reflection.
Alright, but someone with downs syndrome isn't going to understand non-verbal communication. Even if his mother talks with him, he still isn't going to understand non-verbal communication. It's not because he's rude, or his mom isn't parenting him well, or any other reason. It's just the way his brain works, he can't grasp the concept. Because everyone else shook his hand. So they were all, basically telling him, yes shaking hands now is a good idea. Since you didn't even acknowledge him, other then to remove his hand, you communicated that you weren't understand him or "hearing" him. So he kept trying to get your attention.
Is making a small whispering exception that bad? It sounds like, you don't want to be rude to the others by holding up the line to shake hands, or by speaking. Alright, fine. Don't be loud, don't have a long conversation, just very quietly say "Not right now" and move along. Or make eye contact with him and shake your head and then give him the "shh" signal. If his mother was sitting right next to him, she would have had to speak to get him to stop, or to take him outside. I timed myself "I'm praying now, later" took me just over one second and "Not right now" took me less then one second. You can say it as you remove his hand so he understands.