I think one benchmark is to look at the clothing item in question and see if the word "exotic" pops into your head. If it does, then maybe you shouldn't wear it because on some level you are characterizing the item as the Other, strange and foreign, and perhaps you like it for that reason*. You (general you) might think, "Well, I don't mean exotic in a bad way!" but trust me, it can be annoying and offensive regardless. People might think they're being complimentary, but speaking as someone who's been on the receiving end of similar "compliments"...no, thanks.
Sort of related - as an Asian girl, a dealbreaker for me where guys are concerned is if he praises my looks as being "exotic." I have way too much experience with non-Asian guys with "yellow fever" and certain phrases are total red flags. I think the same could extend to clothing.
* I've come across white Americans who like taking on other cultures because they're more interesting. According to these people, "We don't have an actual culture the way you do!" Yes, you do, it's just so dominant that it's everywhere around you and you can afford to ignore it because it's the default.