The problem I had when I lived in a "destination" neighborhood, was not the idea of children coming there to TorT, but that something about the process of being "trucked in" in groups seemed to incite a mob mentality in the kids and their parents.
Part of what makes trick or treat pleasant for everyone, is the neighborliness of it - whether you are geographical neighbors at all. It is a very tiny practice of hospitality. It is an opportunity for children to learn about knocking on doors, greeting people, saying "thank you" and waiting their turn.
When the kids were brought in in large, minimally-chaperoned groups, they felt anonymous - so there was much more pushing aside of little kids, double-dipping at the same house, surliness, yelling and demanding, and generally obnoxious and unpleasant behavior. The grownups either were absent, or did nothing to correct their charges. Instead of exclaiming over costumes and playing along, I spent the whole time correcting, refereeing, checking that the trampled littles were OK, turning away greedy double-dippers, and fielding complaints about the quantity/quality or selection of candy. Sometimes I even had to physically rebuff grabby hands who tried to reach over/under my arms and snatch more than they were offered. It was very unpleasant.
This was not a class thing, as most of the kids who got trucked in were from an equal or higher socioeconomic group, they just lived in areas that were not sidewalked or well-lit, so they came to us.