I think you're fine with what you've been doing, especially because the neighbor has now explicitly given you permission, and before that there seemed to be a tacit agreement. But, I think it's good to know that there are other ways to see the situation--say your neighbors on the other side planted raspberries and you were in the exact same situation, but these neighbors were a bit put out that you didn't ask them before picking the berries--I don't think that would be unreasonable of them.
To me it's like, "Okay, the thing you planted is now coming into my yard. What should we do about this?" Without getting into legal stuff, it seems reasonable to me that you have the "right" to do something about the thing in your yard, but since it originates somewhere else, it is also reasonable to me to consult with the other person involved. And, I think the discussion works both ways--if I see that something of mine is poking into someone else's yard, I should start that conversation with them to see how they feel about it.
In your case, it seems like both parties had an unspoken agreement that you could have the berries on your side of the fence--it's possible they've been secretly resentful of it for years, but then again they've never done anything about it, like say anything to you or trim their bush, so I would've assumed they were cool with it.
In a counter-example, I personally don't actually like blackberries (weird, I know) and in my area they can become very invasive and a nuisance. So my first thought would be to chop off the branches that were coming into my yard. But, I know that cutting off part of a plant can sometimes affect how the rest of the plant grows, so I would go to the neighbors and say, "Hey, this thing is getting into my yard. Shall I chop off the parts on my property, or do you guys want to trim it yourselves/train the brambles to grow differently/etc.?" Hopefully we would be able to come to an agreement about it. But, I wouldn't feel right just cutting the branches without mentioning it to them first, especially if it was genuinely hard for them to see that it had become overgrown on my side.
So in summary, I don't think you did anything wrong, but I don't think the neighbor or your visitors were weird to say what they did. I think you guys were just operating under different assumptions, and fortunately they all came to the same conclusion (that it's okay for you to have the berries). But, I don't think their different assumptions are unreasonable.