I try to sort by genre, but I have ended up mixing them a little.
As the result of being an English major, I have a lot of fiction. I also have a lot of books about authors, and history, where knowing about the history of the time makes a difference. So I shelve all these books together--the books by a given author, the literary criticism about the author, the history books about the time period. It's a logical grouping for me, but the occasional visitor is stumped by this, because I also have a separate section for history books in general.
Cookbooks go in the kitchen, but books about eating and cooking are in the living room, that sort of thing. Because they are my books, I put them where I can find them. Visitors can browse, but they may have to ask where certain books are, i.e. the sci fi books are in the study, hidden by the open door. The etiquette books are in the same bookcase, because that's where they fit, not because I think there is some odd connection between rocket ships and addressing thank you notes correctly.
IMO, the chief thing is to put the books in groups that will help you to remember where they are--if you think of that Brooklyn cookbook as a cookbook, it belongs with the other cookbooks. If you think of it primarily as a history book, then put it with the history books.
(Sadly, my first thought on reading the OP was "Windowsills! Why haven't I thought of windowsills!" This is not going to help my determination to keep the books inside the bookcases and not spilling over onto the floor, the desk, the tables, the attic stairs. . . . )