With regard to car seats, my DH is actually an inch taller than I am, but I have longer legs. He also likes to sit rather close to the steering wheel (even though I've told him and he agrees that it's safer to sit as far away as you comfortably can). So I drive with the seat pushed all the way back, and he drives with it close enough to the steering wheel that I can barely get in.
I also do 90% of the driving in this relationship, and am the primary user of the car (I drive to work, DH takes the bus).
So we both determined that it was eminently reasonable for me to ask him to push the seat back after driving it. I can count on one hand the number of times he's actually remembered to do it, but at least he agrees that it would be nice for him to.
The best compromise we have is that the only things that change in the car when I drive versus when he does is the closeness of the seat to the steering wheel and the rear view mirror. We've gotten the side mirrors to a spot that works for both of us, and we both like the seat back pretty straight. So at least when he inevitably forgets to move the seat, I just have to reach down to the bar under the seat and shove it all the way back.
All that said, I do think it's reasonable, when two drivers share a car and need the seat in drastically different locations, for the driver who sits closer to the steering wheel to at least move it back. I wouldn't personally do more than that (i.e. adjust the angle of the seat back or height of the seat itself), but shoving it back at least means both people can get in the car and adjust it from the seat.
But I think the best solution of all is to have a car that has programmable seat settings, that can be engaged at the push of a button. That is definitely a nice feature, and one I'd love to have when we replace our current car.