I was thinking about how to answer the question in the OP and I realized something interesting.
The only one I ever say "I love you" and "I love you too" to is my Mom. I talk with her every night on the phone and we end every conversation this way.
With everybody else, friends of both genders, and family, I say I love them, either to their faces or not, but it's kind of half-joking that isn't really joking. And it's in the sense of prefacing something I don't agree with that they are doing (not big moral issues or anything, just little stuff).
As in, "I love you guys but why on earth were you wearing your winter coats in the yard, it's July for goodness sake"
or, "You guys know I love you but if you seriously think I am eating that with you, you have had too much Mountain Dew". Or even "I love her to pieces, I really do, but when she came to pick me up from work wearing just her pajamas I thought I would fall over."
So, that counts in a way, I guess. Because I really do mean it. But as a PP said, life is too short. Maybe I should make more of an effort.