Most countries that celebrate a labor day celebrate it on May 1, which was declared International Workers Day (some countries call it Labour Day) in commemoration of the Haymarket incident of 1886, a workers rights demonstration in which four people were killed. This was in Chicago. So the date for the workers holiday around the world was chosen in commemoration of a labor event in the U.S., which doesn't observe that date.
Over the years, some groups have tried to get U.S. Labor Day changed to May 1, in solidarity with the workers of other countries. But that has met with resistance, at least in part because May Day/International Workers Day was observed with great fanfare in the Soviet bloc countries and was considered a "Communist holiday," even though plenty of non-Communist countries observe it. In fact, in the 1950s, at the height of anti-Communist fervor, May 1 in the U.S. was declared "Loyalty Day." It still shows up on calendars, although at least in my lifetime it was never an official close-the-banks-and-schools type holiday.
Some labor activist groups in the U.S. do have some token observance of May Day as a gesture of solidarity with workers in other countries, but the September Labor Day is the main observance.
As far as fashion, as others have pointed out, Labor Day, because of its timing in early September, is the traditional end of summer and time to pack away the summer attire. These days, though, nobody cares if you carry a white purse or wear white shoes after the second Monday in September. Even the U.S. edition of "What Not to Wear" used to display a "White Shoes Permitted After Labor Day" sign.
The above is for informational purposes only -- no political debates please!