I am so glad I'm not the only one...my boyfriend does this, too, and I feel *so* awful for sometimes being slightly...less responsive than I think I should be. For me, the phrase is more of a 'special' thing, even if it is just a 'I'm leaving, love you' thing. And I don't always say it back but I do more than I would if he hadn't said it first, if that makes sense.
The only time it actively makes me annoyed is that sometimes he will do it in response to something I've said that, to me, deserves more of a response than that. So if I am unhappy about something (that he's done or in general) he will say it instead of addressing my concerns. So I might say, "I'm a little worried about money this month, so I'm hoping we can cut down on the extras." And he'll respond with "I love you." It's like...ok, great, but that's not helpful in this situation!