I have no problem with it, it's a fashion thing now. My son is a camo nut (he's three) and he loves his camo pants, sweats, underwear, etc. It's kind of like living with Patton....
I myself have worn camo and I've never heard anyone say boo about it. Your outfit sounds really cute, wear it!