Malcolm Gladwell, “Making the Grade”
on the Comment Page of The New Yorker
September 15, 2003
The most striking thing about the sweeping federal educational reforms debuting this fall is how much they resemble, in language and philosophy, the industrial-efficiency movement of the early twentieth century. In those years, engineers argued that efficiency and productivity were things that could be measured and managed, and, if you had the right inventory and manufacturing controls in place, no widget would be left behind. Now we have "No Child Left Behind," in which Congress has set up a complex apparatus of sanctions and standards designed to compel individual schools toward steady annual improvement, with the goal of making a hundred per cent of American schoolchildren proficient in math and reading by 2014. It is hard to look at the new legislation and not share in its Fordist vision of the classroom as a brightly lit assembly line, in which curriculum standards sail down from Washington through a chute, and fresh-scrubbed, defect-free students come bouncing out the other end. It is an extraordinary vision, particularly at a time when lawmakers seem mostly preoccupied with pointing out all the things that government cannot do. The only problem, of course-and it's not a trivial one-is that children aren't widgets.
Suppose that you'd like to identify and reward those schools which do a good job of improving their students' performance. That's the kind of thing that the industrial-efficiency experts, with their emphasis on "best practices," always said was a sound procedure for companies looking to boost productivity - and the new school reformers have made this idea a centerpiece of their new regime. But how do you measure the performance of a school? It turns out to be surprisingly hard. North Carolina, for instance, instituted a program that every year recognizes the twenty-five schools in the state that record the greatest single-year jump in their students' test scores. As the educational researchers Thomas J. Kane and Douglas O. Staiger have pointed out, that honor is nearly always won by the smaller schools in the state. In fact, the state's smallest schools are about twenty-three times as likely to win performance awards as its largest schools. But North Carolina also identifies its worst-performing schools, and almost all of them are small schools, too. Does that mean that small schools are better learning environments or worse ones? Neither. It means that a lot of the ups and downs in a school's test scores are due to chance factors, such as the presence of a few really good or really poor students in a class, or the fact that on test day a few students may guess right on a couple of hard questions - and the smaller the school, the larger the role played by chance. As it turns out, most elementary schools are small, so it's hard to know, most of the time, whether George Washington Elementary is actually better than Thomas Jefferson Elementary or just - in that year - luckier. California has a multimillion-dollar award system, in which schools win cash grants from the state based on their performance on a 1,000-point scale called the Academic Performance Index. Thousands of dollars in state aid can rest on a one- or two point swing on the A.M., and those scores are taken so seriously by parents that they can drive up local real-estate prices. But the average margin of error on the A-PI. is something like twenty points, and for a small school it can be as much as fifty points. In a recent investigation, the Orange County Register concluded that, as a result, about a third of the money given out by the state might have been awarded to schools that simply got lucky.
Or take something as seemingly straightforward as the "proficiency" standard set by the No Child Left Behind legislation. States are asked to pick their own math and reading tests, and then define as "proficient" all the students who pass them. The bill provides for stringent sanctions against schools that don't meet proficiency standards. But what do you have to score on a given test to be labelled "proficient"? That's not an easy question. One option is to use what's called the "contrasting groups" method. With this system, a large pool of teachers are asked to identify students they believe are proficient in a given subject, those students are then tested, and their grades stand as the proficiency range. Another method, called bookmark, involves ranking test questions in order of difficulty and having an educator mark the question that could reasonably be expected to divide the proficient students from the non-proficient. Then, there's the Jaeger-Mills method, in which educators assign precise difficulty ratings to test questions. According to a recent article by Robert L. Linn, of the University of Colorado, the state of Kentucky gave its middle-school students a reading test and analyzed the results using all three methods. With bookmark, 61 per cent of the students were considered proficient, with the contrasting-groups method, the pass rate was 22.7 per cent, and with Jaeger-Mills 10.5 per cent were proficient. When we say that we want American schoolchildren to be proficient in reading, which standard are we referring to?
One would think that a standard ought to be high enough so that every student has to try his or her best to reach it. By Missouri's standard, for example, just 8.3 per cent of that state's fourth graders are considered to be proficient in mathematics. But if schools are required, under threat of sanction, to raise their proficiency rates annually, it's fairly dear that they have a much greater incentive to switch to a more liberal interpretation of proficiency. Colorado uses a scoring interpretation that labels 79.5 per cent of its fourth graders proficient in math. It's possible, of course, that the children of Colorado are several orders of magnitude smarter than the children of Missouri. The more plausible explanation is that Colorado has found an easier way of leaving no child behind. If you want to develop a class of high jumpers, after all, you don't necessarily have to teach every student proper jumping technique. You can just lower the bar.
This can hardly be what Congress intended. It believed, correctly, that progress is not possible without standards. The truth is, however, that standards are not possible without meaningful systems of measurement, and learning cannot be measured as neatly and easily as the devotees of educational productivity would like. If schools were factories, America would have solved the education problem a century ago.