I posted on the old board about this- our DD is 13 months and since her father and I work full-time, the house is usually a wreck. Again, like you, nothing hazardous or unsanitary, but a lot of things just not put away (like folded clean laundry waiting at the bottom of the stairs, or groceries on the kitchen island that haven't gone into the pantry- stuff like that).
The advice I got at the time was, your guests are in the wrong for not phoning first and making sure their visit was welcome. If they pop in uninvited, they have to take what they get. However, none of the people who popped in on me were rude enough to comment on the state of my home!
If they do comment, you might say, "I know, it's a mess- maybe next time you can call and give me a chance to straighten up for you." Normally, pointing out a rudeness is rude, but in this case, I would say go ahead. You don't have to say it meanly, but there's no law that says you have to take the rudeness of others, especially in your own home.
I stopped answering the door unless I knew someone was coming.