yeah, i do think that if it is too rural some people will think it is out in the middle of nowhere and boring, to long to drive to, nothing to do, no good restaurants, etc. OTOH, If you live in the city other people will complain about traffic, parking, etc. So even if there is no reputation as a "bad" area somebody won't be impressed with it.
That being said, if it truly is or was a bad area I think that there needs to be an understanding. People might be worried about your safety and they are definitely entitled to choose not to visit if they have concerns but you have every right to live there. I do think I would make sure not to make anybody who is uncomfortable visiting the area feel pressured about visiting.