Bananas are not an uncommon allergen and it is very common for one to find the smell of an allergen to be absolutely nauseating. OP, if you haven't, you may want to get tested. I know 3 people with this allergy, all of them found out because the smell of bananas is nauseating to them - for two of them, the allergy is anaphylactic (the third person 'just' gets hives).
In any case, even if you aren't allergic, there is nothing rude or wrong about telling someone politely that the smell of something is making you physically ill. Your boss will probably appreciate knowing, since that knowledge will help him help you do your job a little better (it will be easier for you to concentrate on what your boss is saying if you're not focusing on the gross banana smell).