If there are laws that forbid it, so be it, the OP should follow the laws, but if there aren't then its not rude. If the neighbor dislikes the roaming animals she shoudl write to her politicians and ask for a change in the local laws.
The way I see it, this is not just about who is "right", it is about making sure that Sam the cat stays safe.
If there are no laws governing pets wandering around, then there might also be no laws governing how people can deal with what they see as "nuisance animals"on their property.
If this neighbor is a "wackaloon", then she may stop at writing letters to her alderman. She could very well start setting out traps or laying out poison.
OP, if you prioritize Sam having his "freedom to roam" over his health and safety, then by all means keep letting him out to wander.