With little kids, I think people forget how much preparation and how much stuff you may need to tote along with child. I don't think one week is unreasonable to plan outings, but maybe you could occasionally take them up on activities closer to home that wouldn't be all day. Or compromise, suggest that you check with each other mid-week to see what weather forecast is like, what weekend plans you might want to make, what your availability is.
Another thing I started to do, when my kids were small and sometimes we would get invited to the pool on short notice or want to go to the beach if the weather was nice, is to keep a "beach/pool bag" packed all the time. I would keep bathing suits, towels, sun block, car toys, and snack crackers or raisins packed all the time. Whenever we got back from the pool or beach, I would repack it ASAP and keep it handy so I could grab it and go.
Joy in Virginia