Did you wash it prior to wearing it? There could be a couple of things going on here.
If you didn't wash it, loose bits are coming off.
Linen tends to shed a bit regardless...and wrinkle, which is why I never were it.
Your pants may just attract lint.
Light color lint is going to contrast a lot with dark pants and vice versa.
Personally, I'd suggest wearing it with a different pair of lighter of bottoms.
ETA: I'd also suggest hand washing or washing on delicate by itself or other similar material in a bag or pillow case.