I used to love having an ice maker in the fridge, either just in the freezer or through the door. It was just so convenient.
But then I got into the drinking water business. The water that runs to your ice maker and water dispenser in the fridge tends to sit in the pipes, both the line leading to the fridge and in the rest of the house's plumbing, because the ice and water are 'on demand' and you don't run the water to get cold, and thus fresh, the way you would at the kitchen tap.
Water sitting in plastic piping may leach some of the plastisizers out of it; water sitting in copper piping will leach copper out of it and if the pluming is old enough, lead out of the solder. Here in Ontario, lead solder was banned in 1990 but the lead solder is still available for non-potable water uses and not all plumbers are careful about which kind they grab, especially since the lead solder is easier to work with than the non-lead solder.
If you use the filter option, it will help take some of these things back out of the water. The filter also removes the chlorine, if you are on a municipal water supply.