I'm thinking the crayon type eyeliner on your waterline is most likely the culprit, if it's not your mascara. Sometimes the pencil eyeliners will leave sort of...goopy crayon type black spots on my makeup bag (mostly from the shavings since I sometimes carry my sharpener in there), so I could see something similar happening with your contacts.
But it could also be flaking mascara. Do you wear waterproof or non-waterproof? Sometimes the waterproof is more stubborn than the other kind, although both can be pretty hard to clean off of things.
But I would suggest skipping the pencil liner one day to see if that makes a difference. I'd feel weird wearing any liner without mascara, so I would say that if you still have the issue after skipping the pencil liner, skip that and the liquid liner and see if that makes a difference. If you still have issues after that, it's probably your mascara.
One other thing that's just occurred to me, actually. My mascara is more prone to flaking when it's older and starting to dry out a little. I don't have that problem so much with new tubes because I think they are just more liquid-y, which makes it easier to not overdo it on application. So if it is your mascara causing the problem and you haven't gotten a new tube of it within the past 3 months or so, that might be contributing.