But refusing to shake hands just isn't normal. You may have reasons, but your behaviour is still outside of western societal norms. Therefore the onus is on you to mitigate the other person's surprise at their handshake being refused.
Like, I might decide to sing all the time instead of talking if I had a severe stutter. People would definitely need an explanation of that behaviour, because it deviates from the norm and affects normal social interactions. The behaviour would have a legitimate explanation, but it still wouldn't be normal.