To me, it all comes down to size.
If it's a classroom, with 30 people - no. It can, occasionally, give the impression that you aren't paying attention (even if you totally are). It's also rather obvious in that setting.
If it's a huge ginormous lecture hall with 400 seats and 75% full - I think it's fine. The lecturer isn't going to notice, and there are going to be people multitasking. (At least, that has been my experience with lectures of this size/type).
I'm personally only concerned about disrespect to the lecturer themselves and distraction in a 'small group' environment. I understand what people are saying, but I honestly wouldn't be that concerned about distracting other people during a large open lecture - I'm assuming the knitting would be of a non-distracting nature, i.e., nothing enormous, sparkly, or overly complicated, no long needles, no complex setup. If something very small and relatively unobtrusive distracts the person next to me so much that they cannot hear, how in the world would they ever manage if I were furiously scribbling notes with a scratchy pencil, or furiously taking notes on my laptop... CLICKETY CLICKETY CLICK CLICK CLICK!

That was law school. 125 Clickety clickers in a room all clickety clicking away at the same time. Knitting would have been many times LESS distracting.