I like TootsNYC's approach.
As a side note, one thing I've found helpful is to encourage the person I'm backing away from to find other people to do stuff with/do stuff without me. Like if she were to say, "Hey, I'm really looking forward to New Movie and I think you would like it too. Do you want to go with me?" I could reply, "Oh, I do want to see it, but I just don't know when I'll be able to fit it in. You go ahead and see it, though, don't wait for me." Or about attending a festival together, "I just don't think I'll be able to, but I hope you find someone else to go with, it sounds like you would really enjoy it."
It just seems a lot more positive to me than saying, "No, I don't want to" or "No, I don't have time." It conveys that you wish them well and has that "it's not you/it's me" vibe--"I
can't, but I'm sure someone else out there would love to, because it's not an inherently bad idea." Plus, it explicitly avoids the whole "but I've been waiting until you could go with me!" guilt trip attempt.