I have to disagree that it was a bad idea to tell your boss that you didn't want to take on the new task, or that it's a bad fit for your current skill set. It's easy to get into the position where you're worried that anything you say like that will reflect badly on you, but being assertive about the duties you want has its place. I've always been of the opinion that if you can't say "I have no desire to take on a task that's outside my current job and will interfere with doing the tasks I'm assigned now" then you've got bigger problems than not being seen as a team player.
All told, I think you handled it very well.
Virg