Why can't we wear white after Labor Day?

I've always heard the fashion rule: no white after Labor Day.  But why?

I was under the impression that white bottoms had something to do with it being summer time.  I only thought that it was white pants, skirts, shorts, etc.  Other people think that it includes white tops as well.

Here is an article that explains it all:
Why We Can't Wear White After Labor Day