I am also confused as to why Mary and her husband can't buy out Jane's share of the house. Are they currently unemployed or something? Given that they've been living rent-free and mortgage-free for the last five years, surely they must have some savings?
If it's truly not possible to sell the house, or for one sister to buy the other out, I think Jane should just move in. She should smoke outside and have her boyfriend overnight, if she likes. If Mary doesn't like it, there's actually not much she can do about it (from a legal perspective) as Jane is a co-owner. It's not like Mary can (legally) evict her or anything.