I think you did fine.
There's a guy, about mid-40's in my neighborhood, and there's clearly something off about him. He occasionally takes the same bus I do, although its been about a year or two since I last saw him. When I was seeing him more frequently he would often cut the line - frustrating because occasionally the buses fill and they have a 'no standing' rule, so a line cutter can legitimately mean someone is 15 minutes late as they wait for the next bus. Then I saw him in some yelling matches with people. From what I've observed he's a fairly pleasant person, until he is 'corrected' or told not to do something, then there is no middle ground, he goes right to outrage and indigence. Once he was walking and yelling at two other guys "I'm going to call the cops on you! You can't tell me what to do!" and the other guys, clearly trying to keep their calm responded "no you may never say that to a woman, shut up and leave us alone." The guy in question keeps yelling and gesturing about "you can't talk to me like that! I'm going to call the cops! You're just [insert racial expletive]s, you cant tell me what to do!" The other guys are trying to keep calm and they are still walking - away - and just repeat "shut up and leave us alone." The guy came right up to me and another woman - really close to us - and yelled "take out your phone! Call the police! Those guys need to be arrested!" I said "no thank you", the other woman shrugged. The guy started to yell in my face again "call the police! Call the police!" A big guy from the bus line had stepped up by then and pulled out a wallet with a shield and said "walk away NOW" to him, he backed off a bit but kept yelling about the police and pointing down the block and saying the guys needed to be arrested. The bus then came and we all walked past the guy in such a way as to not allow him on the bus and that was the end of it.
A week or so later, I was on the bus and this guy got on at the next stop, not our usual stop. He came and sat right next to me. I don't know if he recognized me or not, but he started to try to have a chat with me. I didn't even look at him, I just said "I do not want to talk, do not speak to me" firmly and probably with a nasty tone. I wished I could have said "go away" but its a bus, where's he gonna go? (Actually he did get up and move seats.) I know the woman in the seat in front of me thought I was unduly harsh, but I don't care. I am not responsible for that guy's nice day.
And if he sat next to me on Monday, a year or two since I last saw him, and tried to speak with me? I'd be harsh again without hesitation.
Just because someone is special needs does not mean we need to cut them some slack in every situation. Yes cut them slack if they dress a bit weird, or prefer eating everything with a spoon, or speak a bit differently, or whatever. But once they start with interpersonal interactions, that's the line for me where I don't need to put up with it. Special needs does not mean "nice" or "harmless", and I am under no obligation to speak to anyone just because they approach me.