This is one of those things where servers can't win. If you wait for the signal (which most people don't know), you're going to have irate customers wondering why you're not taking their dirty plates. If you always ask, some people will be mad for how intrusive you are. If you take them without asking, some people will be mad that you took them too soon, or didn't wait for the signal. The only mostly-sure way to know if people want their plates taken is to look for the ones put on the end of the table. Usually that means you're supposed to take them. Usually. I did that once (there was a lettuce leaf pushed of to the side, but it was otherwise empty), and the lady slapped my hand and said, "No, no, no, no. NOT DONE." Seriously. It was so mean that it was hilarious.
I think the only time you can really expect servers to go by the silverware signal is if you're at a pretty nice restaurant. Otherwise, just tell them politely not to take plates until you're both done.