I guess either method is etiquettely acceptable. Personally, if I'm in the bathroom, I'd much rather someone just try the door, realize it's locked and wait, then knock and force me to have to yell out that I'm in there. I also always just try the door handle in public restrooms...though, I guess now that I think about it, if I'm in someone's home (like for a party), I always knock at the bathroom (I've noticed that for some reason, not everyone locks bathroom doors in homes and after walking in on more than one person, I've learned my lesson).
Well not all home bathrooms have locks on the door. Most of the homes I've lived in have not.
Maybe it's a regional thing, but I've never lived anywhere where the bathrooms in the home, apartment, dorm, etc. didn't lock, unless it was broken, of course.
I think it is a regional/cultural thing. Where I grew up in the USA, some home bathroom door had locks, some didn't. The norm was to knock on the door unless you knew for certain that it was unoccupied. Plus, many people leave the door at least partially open when the bathroom is unoccupied, so often the closed door itself indicates that the bathroom is likely occupied. There, in someone's home, you can't count on the fact that the bathroom door will have a lock at all and especially can't count on the fact that the occupant will lock the door. Trying the door handle instead of knocking is likely to result in you walking in on someone.
OTOH, when I lived in Germany, all bathroom doors had locks and they were kept closed at all times, whether occupied or unoccupied. There, you could count on the fact that an occupied bathroom would be locked. However, all of the public restroom stalls there had the built-in vacant/occupied indicator in the door handle, so there was no need to jiggle the handle.
[Off Topic] I also found that the people I worked with in Germany didn't expect people to knock and wait for a response at closed office
doors either. In my workplace, if you needed to go into someone's office, the door would be closed (no one left their office doors open), and the standard procedure was to walk right in. I knocked lightly as notice I was coming in, but I don't recall whether the Germans in the office even did that. This applied even walking into my boss's office. Administrative offices at the university seemed to follow the same principle. Not realizing this, I think I and another American exchange student irritated several people by waiting for a response to our knocks before entering. I'm not sure whether this is a general rule in Germany or it was local--door-knocking customs unfortunately were not covered by our exchange program briefings!
ETA: If anyone is interested in comparing regional variations in door-knocking customs, I just started a thread here