Agree with PP, but I would be very careful not to use any rude language like "shut up"
He also has no authority to summon you into Boss's office for a private conversation. If he starts in on you, leave. Just walk away.
Some possible phrases:
"If you want to say something to me, you can say it out here. I want as many witnesses as possible."
"I do not report to you, I report to Boss."
"You should know I have talked to Boss about you interrupting my work. I am logging how many times you interrupt me and I will be reporting to Boss about it."
"If Boss is not happy with my work product, he can talk to me directly. Whether you are happy with my work product is irrelevant."
"No. I will not interrupt my work to have an irrelevant conversation with you."
Do not JADE (Justify, Argue, Defend, Explain). You don't owe him an answer to any question or a response to anything. Frankly, if you can bring yourself to ignore him completely, or dismiss him with a noncommittal noise and turn away, that is the strongest position you can take.