I tend to sort, much like OP, and toss the couple ones. I get uncomfortable seeing the couple-y ones and see no reason to hold on to them. I know some people think its weird or wrong or something to throw away photos of people, but I don't. I'm not throwing away the person or the memory, simply the relationship part; sure it happened, but I've moved on.
Basically I focus on remembering the broad memory and edit out the "ex" detail. My trip to Ireland, not my trip to Ireland with ex. The trip I what I want to remember, the ex being there is merely a detail. Millenium New Years with my friends in the cabin, not Millenium New Years with ex and friends - he was merely one of many people I interacted with that night.