I guess there's a time when hosting is not to our tastes, a time when it is rude, and a time when it is just plain old poor hosting.
OP, as I read your post, I was picturing the bride's sister as someone who was looking for the spotlight, thinking she was the life of the party, and craving the microphone. IMO, unless she was quite talented and the sound system could support a performance, I don't think inflicting her on the guests was a good idea, and was probably tacky, but not rude. I also think a not-super-close guest is not obligated to stay for the duration of the reception; no need to endure a party not to your liking. I'd say that was not rude, but simply not to my taste.
The disorganized aspect of the hosting sounds to me like it was just poor hosting. I suppose you can use that memory to remind you to compliment the next host you encounter who put in the time and effort to pull off a great event?
From my read-through of the situation, I think the photograph thing was rude. It sounds to me like the whole reception came grinding to a halt for it. I get the impression that all non-family guests were there with no music/dancing for entertainment, no food (dinner was over), nothing to do but wait for the pictures to be over. Sure, they could mingle, and many lovely receptions don't have music, dancing, or food anyway. *but* to have all of those things and then bring them to a halt for nearly an hour of a family-only activity would convey a feeling to me that either the reception is over or my hosts want us to just wait around for a while so they can do some stuff without us, then we'll continue with the party. If I were a non-family guest, I'd likely choose that time to go home, and be greatful that I wasn't a fringe-family-member who was expected to wait for 45 minutes for the one big group photograph they wanted me in.
I know there are possible explanations of circumstances where this was an unplanned work-around for something that went wrong. We don't know for sure. They were likely entertaining according to their tastes and preferences, rather than your (or my) tastes. There's nothing wrong with that, and there's nothing wrong with you going home early to enjoy Sunday night in your own way!