I, too, have been trying to hold my tongue on this, but I guess I feel a need to speak up -- blame my Challenge and GT (gifted and talented) classes if you must, but it's probably more innate than that.
Challenge was the name for our 3/4 day, once a week separate class in elementary school. You had to take several tests to get in, and they were basically IQ tests, with an interview component. The idea was seeing whether you had the potential to think creatively about a problem--whether you needed an extra challenge beyond the classroom. In our grades of about 6 classes each, there were usually 5-15 kids who qualified. The Challenge activities were basically a semester of enrichment around a single topic. We did word problems and logic puzzles, drew or wrote creatively around the topic, and sometimes did more math- or technology-related activities. It was such a relief to be around those kids for a day a week!
In upper grades, we were mostly integrated with "honors" or "AP" kids (other kids who were smart, but frankly not as weird as GT kids-- we generally just looked at things a bit more sideways), unless there were enough of us interested in a subject to get a whole class to ourselves. Sometimes, the teachers would stretch a topic to get a more "GT" way of thinking about it, but often we had the exact same assignments. We either rose to the occasion, or chose to go back into the mainstream classes.
What was the effect on my personal development? Not much! I certainly didn't feel empowered by it, or oppressed by it. It was (rarely) a reminder of my ability to achieve if I bothered to try, but I wasn't afraid to stream back into "regular" math for my upper grades (I loathed geometry). My peers are doing well, or not. Many of them are doctors, PhDs, lawyers, engineers and teachers. Only a couple didn't go to college. A few have other "less prestigious" careers. My siblings were GT, as were most of their friends, and I haven't heard of anything bad happening because of the label there, either (aside from one man with schizophrenia who talks about being a genius ... but that's another conversation entirely).
I've never heard them complain about their label, or use it as an excuse to act weird. I am very grateful for my classes--most of my best friends were "geniuses."