It's lack of clarity aggravates me when used outside of specific situations. Refusing a top-up on a glass, 'I'm good' makes perfect sense. But the expression and use is fairly new (at least to me) so when people used it to, say, tell me they didn't want anything else with their order at work, it confused me at first. It's a slangy expression that I think isn't immediately obvious. I also hear it's cousin 'I'm OK'. I have also heard it spread to other uses of 'no', for example 'do you want to come over tomorrow and watch a movie' 'I'm good'.
I don't want it to become a shorthand for any sort of refusal...I'm only imagining. 'Will you marry me?' 'I'm good, thanks!'