That is a bit strange, only beacuse the arrangement is only to let the dog out to do his/her business. Had you also asked them to play with the dog, or keep it company, then no, but I'd be a bit annoyed to come home to that!
I used to pet sit for neighbors in HS, and some would specifially ask me to stay and play with them, watch tv, etc. so the pet wasn't lonely. Others, it was just come in, feed them, etc. and nothing more.