So... back on topic and speaking of violence.
Be very very careful in these situations. If the guy (I say guy, because let's face it, it usually is) is testing your boundaries and starts to do so physically - he may very well be a predator looking to attack you for either non-consensual scrabble or plain old robbery.
Putting in earbuds or otherwise looking distracted are generally ways to clue someone of this type on to the fact that you're not aware of your surroundings. This is the kind of thing they're looking for in victims.
Similarly, being too polite and letting someone into your personal bubble tells them that you will not fight back. If your creepymeter alarms are going off, be the prickly person who tells strangers to get lost. It's safer than being the friendly person dragged into a ditch and murdered. You don't have to be nice to strangers who don't respect your boundaries. You don't have to be nice to anyone who doesn't respect your boundaries.
Don't act scared, act alert. If they keep after you, reach into a purse or pocket as if going for pepper spray or a gun and leave your hand there. Duck into an open business with security cameras if possible. Above all, be safety-minded instead of just etiquette-minded. Sure there will be false alarms sometimes, but that's much better than letting them close and having something awful happen to you.