I'm trying to say something, and I can't quite get it right... Grrr, I hate that. I just want to throw the idea out there that "not free to go to the party" might just mean you plan to stay home and read a good book, clean the bathroom, have a quiet dinner. Those are the "plans" you've made for the day, and if someone calls to say Grandma's party starts in 5 minutes, you can still say, with perfect honesty and politeness, "Oh, so sorry, we already have plans. I wish you'd let us know sooner."
I don't know if that applies exactly to the OP here; but some people feel like they must attend "something" if invited when otherwise they were going to do "nothing," even if they don't really like the "something" or its last-minute nature is inconvenient to them. And some people feel better if they can decline by saying, "Oh, we already have plans," to divert suspicion that the real reason is, "I just don't like you or your activity."