Here's a little more to the story:
He usually stops by when I'm in the middle of something, I have a very busy schedule. I don't really consider him a friend, he's more of an acquaintance. He weirds me out a bit, honestly. I live alone and he's 18 years older than me. I have a boyfriend, but he lives 40 miles away.
Not last night, but the time before that, he came over and freaked me out a bit. He said "Are you mad at me? Because you're usually very nice and last time I came over you seemed annoyed with me." He was very spazzy when he said it, he also showed up unannounced when I was rushing to get out the door. I leave my lights on when I leave to I appear to be home and a few months back, he told me "Your lights were on and I knocked, but you didn't answer." Well duh, wasn't home or wasn't answering the door. He also told me yesterday that I look tired. Not very polite.
Next time he drops by, I'm going to say exactly what Nya said "Oh Neighbor, I wasn't expecting you. I'm afraid can't visit with you right now." and close the door. He doesn't come over to borrow things, he just wants to talk. I think I feel even less comfortable because I'm home alone. I know I could take him on if he tried to attack me unarmed, but I just don't feel very comfortable with him being in my apartment. I joke with my friends that he's a serial killer because he's often too friendly. One time he asked me how I profiled him when we first met, I flat out told him "I thought you were a serial killer." It's still a possibility
I'm a crime buff, I'm a little extra paranoid (which makes me less trusting in general). I would never just show up unannounced unless I didn't have their number (this guy has my number), I would hate to interrupt someone in an intimate moment or when they're napping. In my eyes, it's just rude unless you're really close to that person and know it's okay.