There are also some family habits you can install that won't require any stepping-on-of-parental-toes, but would streamline a few things, such as:
The night before (perhaps as everyone is cleaning up after supper), assemble the breakfast plate, and store it covered in the fridge. Then, no matter who is up with the 4yo, it's as simple as "open fridge, remove breakfast."
Help 4yo get into a habit of Up, Eat, Dress, Teeth, Play. It'll be less to do if Dad does have an abrupt transition from "getting ready" to "out the door."
Likewise, make it a habit to have shoes/coat/bag packed and ready to go by the exit the night before. Less abrupt transition.
I wholeheartedly support having both parents parent equally, as you're doing, and I love that you're not wanting to dictate how everything ought to go all the time. But, individual personalities have different strengths, and pointing out how best to use his strengths in parenting (for instance, I'll bet he's fantastic at spontaneous fun outing or play stuff!), while meeting the basic needs of the little one, is not dictating. It's making sure they don't freak out at one another.

I've a dear SIL who has an early-riser son. Seriously EARLY-riser, 5am every morning. And he wakes up STARVING. And he's crabby until he has eaten, every single time. But she's not always quite up to Being the Mom at 5am. And she's never hungry right off the bat. So he just starves and starves and get crankier and crankier, until there are Major Explosions in the household, and being barely 6, he has a hard time getting past that and into his day.
Enter: staying over at Auntie Liz's house in the summer. I do NOT get up at 5am. I also am not able to eat first thing (I usually need to be up for a few hours before I can handle anything solid.) But, I've seen him melt down, and I do know it's because he's hungry--he lets everyone know.
But, I don't want to be bossy about my SIL's parenting, because she's really doing a good job overall, and loves her kids tremendously.
So instead of announcing any Grand New Routines, etc, I just sent him out to the tent with the other kids as dusk descended, and handed him a little airtight container as he went past.
"What's this, Auntie Liz?"
"That, my dear little man (okay, actually I call him Stink-Face, very affectionately), is your morning food kit. There's some fruit, and an applesauce muffin, and some granola with seeds, and half a peanut-butter sandwich, cut into triangles. So when you wake up and are hungry, just eat some of it until you feel good, and have some water, and then you can read books until the other kids wake up."
And my goodness! What peaceful mornings we had! He's a clever, delightful kiddo... but he NEEDS a routine that makes immediate-food possible and under his own control (at 6, he can do this; at 4, I would have gotten up and set food out for him in the AM, then snoozed on the couch.) SIL hasn't made it a habit just yet, but she is seeing a difference in him on the days she does this, and it's working... without me ever having to feel bossy or intrusive or rude.