OP - may I ask how many people were going to go to lunch? I have to imagine it was just more than you 3 - because that wouldn't be a big group. If there were more than 3 of you going; how did the person asking you to lunch deduce that you didn't want to go because of the other coworker?