So a person with downs syndrome, still a person. Just a person whose communicates differently. So subtle clues just aren't going to work. This isn't because no one has taught him. This is the way his brain works. It's different from your brain (different, not wrong), and his mom has probably gone over at length with him appropriate church behavior. But what he can recall is "People will want to shake your, shake their hand, this is a friendly sign we use to welcome people at church", it's a general overview. Other people were shaking his hand, so he figured this was hand shaking time (everyone was standing up, like when everyone stood up and shook hands earlier, so hand shaking time, awesome).
The best way to handle this, is by directly speaking to him. Remember, he's a person. I would have shaken his hand quickly and then said "I'm going to pray quietly now, this is a time for quiet prayer", so he understood why you were done interacting with him. Or, if you really didn't want to shake his hand, saying "I'm praying right now, we'll shake hands later", to let him know what was going on. By remaining silent and not acknowledging him, what you were communicating to him, was that you didn't notice him or understand what he wanted. So he kept trying to let you know.
In the future, speak to him. Not in a scolding manner. If he tries to shake when you don't want too, just let him know now isn't a good time to shake hands.