I grew up on Army bases around the US and the first poppy I ever saw was when I spent a year in England when I was in my 30s. So I think the poppies are somewhat regional in the US. I did go to the Veterans' Administration website for more info, and they state that in the US, poppies are properly worn on Memorial Day, which is the US day to remember our fallen war dead. Veterans Day, although it is celebrated at the same time as Remembrance Day in other countries, is a day to celebrate our living veterans, and therefore the poppy is not appropriate.
My grandfather, a WWI veteran, did make me memorize "In Flanders Fields."