I do all the grocery shopping and sometimes the bagger will move aside items that are light and/or delicate. When that happens, it's really difficult to resist the urge to move it right back, because about a quarter of the time the bagger completely forgets the moved item (since it's hiding behind the bags that haven't made their way to the cart yet) and I end up bagging it myself. It's aggravating because I'm not all that particular about what bags things go in, as long as they're in a bag. That's my completely long-winded way of saying, perhaps there is a reason people are moving the groceries and it has nothing to do with you! (Unless you're the bagger who keeps forgetting to bag things!!
I agree with O'Dell, just start narrating your actions to the more antsy customers. It's more tiring to have to talk all the time, but it's easier than arguing all the time!