Okay, this is going to sound wacky, but... do you remember what book you want by how it looks? If you know you're looking for, for instance, a murder mystery with a black leather cover or a cookbook with a soft green cover, you could try grouping them by color and then by some other method (topic, author, title, etc).
I've seen several pictures online of a bookstore that has tried that approach, which seems like trouble, but in your own personal collection, if that's the way you remember books, that might actually be manageable. I would skip that approach for series, though, so they're all together, even if they aren't all bound the same way.
My parents' books are grouped by type, mostly, all text books together, all coffee table books together, all children's books together, but then within those groupings, I don't think there's much order.