You know, I kind of wonder if this assumption is actually sort of complimentary. There are many smokers whose habit is obvious - their clothes and cars permanently stink, and they are continually rushing outside for their next fix. This is the case even with many of the nice, polite people who smoke! Then of course there are the even ruder breeds who smoke in people's faces etc.
I know smokers who aren't like this, who smell nice (not that yukky smell of tobacco inadequately covered with breathmints either!) and who many people don't know smoke because they are subtle about it.
If somebody is surprised to hear that you're a smoker, it might be that they are impressed that you manage to do so in such a considerate manner, not that they are thinking less of you. If I were a smoker, I'd prefer to be in that category and know that my efforts to be polite about it are working!
But as a non-smoker, my interpretation may be off. :-)