Both the restaurant AND the previous table are at fault.
On one hand, your party should have never been brought back to the room unless it was clear, clean and ready for your party. Sounds like a host/hostess screwed up. That should be addressed with the MOD.
OTOH, I have seen scores or customers that come in, eat, pay, and then camp for hours at a time, either for showers, birthdays, business meetings, etc. In my experience, the servers are told to come back repeatedly and ask if they need anything else, clear as much of the table as possible, and generally attempt to subtly urge the customer to vacate the table. Sometimes this works, sometimes not.
If the people at the table had observed the setup for your party, and then saw you brought back and waiting, then it was also very rude for them to continue to use up space after the meal and payment. When one goes to a restaurant, it is to eat. Once eating is done and payment is rendered, it is not a free place to hang out for long periods of time.
The worst one I had personally was a couple who requested a big booth (we had 4 top booths and 6 top booths) as it was the end of lunch (2 pm) and we had a lull till 5 pm, they got the 6 top. They ate, had dessert, etc. and paid by 3:30. They were still there at 5 when I went home (I paid another server that was coming in 5 bucks to reset the table, otherwise I would have had to wait for them to go). I heard later that they hung out till almost 7 pm, well into a Friday night dinner rush, chatting. Not only did this prevent people waiting from eating, it cut into the incoming server's section, and cut into her pay. In a restaurant where the average table is sat 3-4 times a night, and the average tip per table is anywhere from 7-15 dollars and up, that can be a chunk of cash.