Having your wedding when/where/how you want to have it is not rude. Changing the wedding location to better accommodate your guests is above and beyond (sacrificing your desires for theirs). Knowing some people may not be able to attend your wedding (which could happen regardless what date/time you choose) is not rude.
Guests complaining about what you are offering is rude. Guests demanding the wedding location be moved to their city to accommodate them is rude. Guests expecting based on the time of the wedding (afternoon-dinnertime) that there must be a full fancy dinner and dancing and booze? Rude.
I think a late-morning ceremony with brunch/luncheon to follow sounds wonderful! Sure, some people may have to take a day off work to attend, but as other posters have said, that's often the case with Saturday weddings as well. OP, you've saved your guests the travel time, airfare, hotel, baby/dogsitters by moving the location to their city. Because their city costs significantly more, you've adjusted the time so you can still afford to entertain your guests as you'd like. Sure, some people may decide the event is not worth the inconvenience, but that's on them.
OP and her fiancé are hosting a party. Their invitations inform invitees what to expect. Invitees then decide whether or not to come to the party. I wouldn't dream of telling a bride/groom their party is not up to my expectations, and demand they change it for me. It's their party, their celebration, their wedding, not mine. I can choose not to attend.
OP, don't worry about pleasing every last person. Decide what's important to you and have your celebration. The worst thing about weddings I think, is all the stress of planning it, which is amplified when you try to make everyone happy. I hope you can make your plans, and when the day comes, relax and enjoy yourselves, as the day is about you, celebrating the start of your new life together with your partner.