My cultural background is Indian and I have no problem at all with Westerners wearing saris. Various people in my family, myself included, have taken English friends to Indian areas of London to shop, etc. Saris aren't inherently formal - it depends on the type of sari. Some are quite plain, some are highly embroidered/decorated (some in between), some are more formal fabrics (e.g. silk) others are washable (e.g. cotton), it's like Western clothing really. Similarly salwar kameez are not inherently less formal, they just originate from different regions of India.
The only thing that does sometimes annoy me is when people wear them incorrectly - and, even then, it's not so much that I think it's rude or anything, more that I itch to rearrange it.