This is a very interesting thread for me. Where I live, just about all fences are on the shared boundary line (by law) and 90% of the time the costs are shared by both property owners, so it's unlikely one neighbour could, or would, just pull the fence down. I actually can't even envisage a property without some sort of fence between it and the next one, even if it's just a row of garden beds, as I don't think I've ever actually seen this.
That aside, I agree with pp that OP should give the neighbours notice that the fence is coming down. If I were one of the neighbours, and I owned a dog (or other pet that can wander around at will), and the fence was there one day and gone the next without any warning, I wouldn't feel any remorse if said pet got into your yard and did damage, as I wouldn't have been given the chance to prevent if from happening
eventually I would be selling, and a fence would be a plus for the new owner.
I wouldn't even consider buying (or renting) a property that didn't have a fence around it on all sides, bar perhaps the front. In fact I probably wouldn't even get out of the car to go and look inside. I like a clear boundary that says 'this is mine and that is yours', and if it's a six foot high privacy fence, all the better.