Okay, I understand that by the time you got to the feet-on-placemats thing he had already done a lot of other stuff, but that totally squicked me out and I can't believe you stayed silent! You are so busy worrying about "nagging" that you aren't teaching him your house rules and are instead sending him mixed and inconsistent messages.
Does he get defensive and accuse you of nagging? If not, then don't feel about about telling him The Way Things Are In Your House. This doesn't have to be in lecture form. Just point out the obvious - he isn't doing things that can be defended. The sofa is not a napkin and the place mats aren't foot rests - say so!
If you are worried how you will sound, don't scold but keep it light and maybe try what I would call the "gosh you are (almost) so smart" method: Act like you are giving him the benefit of the doubt and think he either knew the right thing all along and just forgot, or is almost to the A-plus behavior with a gentle nudge. So you can say things like:
"Hey Bro, I see you were aiming for the shoe area and missed - thanks in advance for moving your sneakers so no one trips."
"Oooops, woah Bro - shoes go on hall mats but feet don't go on placemats - feel free to move those before you put your feet up 'cause I really don't want to taste your socks!"
"Hey buddy, you don't need use the whole sofa - just let me know you need a napkin, they're easier to throw away anyway amiright?"
*head tilt and asked with a genuine and amused smile* "Um...what were you planning on doing with that coaster, if you weren't going to put your drink on it? Be a good sport and save my table, 'kay?"
If you keep it light instead of silently seething, sighing, eye-rolling, etc. then he shouldn't feel nagged. Give him a fun little push instead of wagging your finger at him. Also, if you focus on how much you appreciate him doing things right, he should feel validated instead of put-upon. Call it "positive reinforcement training" instead of "how to get your sister of your case."
Forget the idea that you shouldn't have to say anything. We all have our clueless moments. Don't assume he knows or complain he isn't psychic.
It's your house and you know the way you like it - it is up to you to communicate your wishes to your clueless guests - whether they are related to you or not.