My parents pulled my older brother out of school because he wasn't being challenged enough. He had at least one great teacher who found extra projects for the students who had mastered the current material, and that year was great. However, his last year in school, not only would the teacher not find anything to challenge students who had already grasped the material, she actually suggested he be held back a year when he acted out due to boredom. My parents decided that was enough, and they would handle it themselves. They also decided to homeschool me at the same time. We both thrived. My mother worked fairly closely with the local school district: they lent her some of the same textbooks they used, and we went to the school to take the same standardized tests every year to demonstrate that we were academically on track, etc. When we started studying topics our parents couldn't teach (foreign language, lab sciences, calculus, etc.), we were able to enroll early at the local community college. (I'm currently in grad school and my brother has a PhD, so we definitely didn't suffer academically, nor were we unable to integrate into formal schooling later.
As far as socialization goes, we were part of youth groups at our church, frequented a public library with youth events, did art camps and sometimes other summer camps, attended a home-school group that met once a week, had friends on our street, etc. I've never been a social butterfly, but it wasn't because I lacked the opportunity to hang out with other kids.
Other benefits mentioned by PPs are spot on: If there was something fun and educational going on, we could take off without worrying about the schedule. If one of us got interested in a particular subject, Mom could let us go to town at the library and turn it into a project. We ended up socializing with a lot of wonderful people from other generations, not just kids our age. I also had a lot more free time than my peers in school, because once I completed my schoolwork for the day, I was done.