It takes quite a lot for me to not finish a book, in part because I'm kind of choosy about what I'll pick up to read in the first place. Often, the bigger test for me is whether I have any desire to read the book again, since there are a lot of books I've read once and never had the slightest inclination to reread. But all the books I actually like, I have at least the desire to reread should I come across the book again (since I do read a lot from the library, and so don't own all the books I've enjoyed).
But for me, the number one thing that makes me stop reading a book is when I stop caring about what happens to the characters. That can either be because I don't like them, the plot is boring/nonexistent, or because I just don't give a good d*** about the people in the book. I used to keep reading such books anyway, but I finally threw up my rhetorical hands and gave up when I was assigned Return of the Native by Thomas Hardy in high school. I just could not make myself finish that book, even though it was required reading, because I realized that I hated all the characters and wished they'd go jump in a lake.
The number one thing that makes me not want to read additional books by the same author is messing with me at the ending. Thomas Hardy is the standard bearer for this for me, because of Tess of the D'Urbervilles. That was another book that was assigned for a class I took in college, and I did finish it. But I wanted to fling the book out the window when I got to the end. It's one of those books where the basic plot is "bad thing happens to heroine through no fault of her own", then it gets a little better, then worse thing happens, then it gets a little better, then even worse thing happens, until finally by the end, the heroine is left in her misery. And in the case of Tess, the "hero" (who I thought was a thoroughly annoying character) gets all virtuously mopey about said misery even though he was directly responsible for causing a good chunk of it.
So given Tess and Return of the Native, I am never picking up another Hardy book ever again, and nobody can make me.
Another thing that will make me not want to read a book again, and be wary of a particular author, is plots that depend upon someone being Too Stupid to Live. Or where basically the whole plot depends upon a situation that would be resolved after a 5 minute conversation (which often goes hand in hand with one or more of the main characters being TSTL). Julia Quinn is a romance novelist who I actually think is quite a good writer, but a couple of her earliest books follow that model. It's fortunate for me that I read those earlier books after reading (several times) most of her later body of work, which is quite enjoyable and features books that don't depend upon that particular plot device. So I can still read and enjoy other Julia Quinn books, and just pretend that the ones requiring that particular plot device don't exist.
Beyond that, I have a pretty high tolerance for language and grammar issues, if the plot is otherwise interesting and they don't make the book unreadable or impossible to understand. I'm fine with predictable plots if, again, they are interesting and I like the characters. I actually like my happy-ever-after endings so much that I really do read a ton of romance novels, just because I know that's how they'll all end. Some are pretty awful (and I read those, when I read them, more because the Scottish-Laird-Pirate-with-a-heart-of-gold or whatever character who is tamed by the spunky-but-nurturing heroine that shows up a lot in historicals can be quite hilarious). But some romance novelists are actually pretty good writers, and mostly produce very enjoyable books (albeit with rather predictable plots).
