At my workplace there is a women's and men's restroom. Both have the exact same equipment inside. No stalls, just a room with one toilet and one sink and a door on the outside. People use whichever bathroom is available. In fact, while they are labeled separately, only the men's restroom is fully ADA compliant, so I'm not sure that anyone could even demand that the restrooms be single sex.
So basically, if only one person can use the bathroom at one time, I don't see the big deal in the men using the women's restroom. In fact, I don't blame them at all given that their only other choice is to use a restroom where the bathroom door won't lock. I would have a different opinion if there were multiple stalls, but in this case, I think it's okay.
Edited: I didn't realize that there is a lockable stall inside the men's restroom too? If that's the case, then they should probably use their own bathroom, but again, them using the women's wouldn't bother me personally.