I'm a huge fan of costume, clothing, and textiles, and I've got quite a collection of clothes from various places. Kimono- both cotton yukata and elaborate wedding robe, a Chinese Qipao , two sari, both a simpler everyday one and a stunning green and gold brocade one, and a robe from Ghana.
I have been known to wear those out in public, and I don't get flack for it. I think it's because of how I carry myself- I don't consider the clothes to be costumes and don't wear them as such. I wear them like regular clothing. And I don't act any differently than I would in regular clothes.
When it comes to sari, I love how they look good on so many body types- unlike many western styles which only suit certain figures. Anyone from a rail-thin model to a generously sized woman can look awesome in one.
Now, that said, I've decided not to wear some things, like the string of Krishna beads I saw for sale at a local import store. I read the sign next to them explaining their purpose, and the things that a wearer would have to do to show the beads proper respect, such as practicing a strict vegan diet.
Needless to say, I decided not to bring those beads home with me, as I would not be able to meet the religious obligations.
Likewise, Native American/First Nations dress and regalia is a sensitive area- Outsiders wearing war bonnets, or misusing other objects or regalia is a real issue, and many Natives have asked outsiders not to do that.
So I think that as long as you don't treat the clothes like a costume and don't put on an act while wearing them (trying to act faux-Japanese like some obscessed Anime fans do, for example), I think it's fine. Cultures borrow from each other all the time.