Ugh, that's hard.
Can you couch it as personality, not bad?
I can imagine...
"friend, you know how there are people you adore but who you should never live with? like how I get along so much better with my sister when we don't share a roof, but I want to beat her with her own arm, after I rip it off of her, when we share a house for more than 1 day?
I see the potential for that sort of personality conflict between you and the other housemates. Since this is their house now, I just don't feel right imposing another room mate when I forsee the 3 of you together being oil and water"