How We Measure Up

As if coordinated to provoke headlines, top executives at three of the nation’s leading technology firms recently issued bleak appraisals of the American education system, criticizing especially how American students are taught science and mathematics. Microsoft Chairman Bill Gates minced no words at a summit of the nation’s governors: until high schools are redesigned, he declared, “we will keep limiting, even ruining, the lives of millions of Americans every year.” The chief executives of Intel and Cisco Systems shortly followed suit, suggesting that America’s lackluster schools will increasingly force companies to look overseas for talent.

Of course, these concerns are hardly new. But the somber prognoses from the heights of high-tech have added high-profile urgency to recent press reports about the declining performance of U.S. students in science and math compared to other nations, and the potential rise of China as a technological and economic superpower. Leading U.S. media outlets have featured major stories on the consequences of China’s rise for America’s future, like the recent Newsweek cover story by Fareed Zakaria appealing for a “massive new focus” on science and technology in the U.S., lest America “find itself unable to produce the core of scientists, engineers and technicians who make up the base of an advanced industrial economy.” In such a media atmosphere, one could be forgiven for having concluded that the United States is drifting unawares into an educational backwater while the rest of the world paddles furiously past it.

The truth is more complex. Cross-national studies of scientific and mathematical ability, interpreted rightly, tell a complicated story, giving reason to question how well the tests measure America’s real educational standing in the world. The two tests cited most frequently in press reports are the Program for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS). PISA, undertaken by the Organization for Economic Cooperation and Development (OECD), most recently spanned 41 countries and tested 15-year-olds on mathematical word-problems. The latest TIMSS, in 2003, comprised more traditional, textbook-style math and science problems and was administered to fourth- and eighth-graders in 25 countries by an international team of researchers based in Boston and Amsterdam. The Department of Education funded both studies in the U.S., with help from the National Science Foundation.

Both tests have repeatedly been invoked by sensationalists seeking to cast the United States as unprepared for the high-tech, global economy. When the latest PISA results were released toward the end of last year, for instance, the Christian Science Monitor ran with the headline “Math + Test = Trouble for U.S. Economy,” and concluded that the study’s emphasis on “real-life” math skills makes it an accurate and “sobering” predictor of students’ performance in “the kind of life-skills that employers care about.” Federal officials expressed concern about the test results, too. “If we want to be competitive, we have some mountains to climb,” said Deputy Education Secretary Eugene Hickok.

To be sure, the results of neither TIMSS nor PISA should send American educators and policymakers rushing to the champagne. In most math areas tested by PISA, the gap between the average U.S. student and the average student in the highest-scoring countries — often Finland, the Netherlands, Singapore, Japan and Hong Kong — was roughly equivalent to the gap between the United States and low-scoring countries like Uruguay or Mexico. Where 44 percent of Singapore’s students reached the TIMSS “advanced international benchmark,” only 7 percent of U.S. students did. And, in general, the longer students had remained in the U.S. school system, the worse they performed relative to their peers abroad.

The first question that must be asked of such broad results, however, is whether the tests themselves accurately represent the countries’ student populations. International surveys such as these are not given to every student in each participating country; the tests’ organizers pick out statistical samples that are supposed to represent each country’s entire student population. Even so, schools — especially in the United States — sometimes decline to participate in the tests, potentially skewing the sample. As far as accurate sampling is concerned, early incarnations of the tests were not encouraging. In the first TIMSS general achievement test, conducted in 1995, only 5 of 21 participating countries met the study’s guidelines for conducting representative samples. While most countries participating in the latest studies have dramatically improved their overall sampling, the United States remains a notable exception. Only 73 percent of U.S. students chosen to be sampled were actually tested, a figure below the “minimum acceptable” rate of 75 percent. In most other countries, that number was well over 90 percent. If the omitted 27 percent of U.S. students were even slightly above or below average, their exclusion casts serious doubts on the accuracy of the U.S. sample.

The studies also inevitably confront large differences between countries’ school systems. “In Cyprus, students taking the advanced mathematics test were in their final year of the mathematics and science program; in France, the final year of the scientific track; in Lithuania, the final year of the mathematics and science gymnasia; in Sweden, the final year of the natural science or technology lines; and in Switzerland, the final year of the scientific track of gymnasium,” Professor Iris Rotberg of George Washington University wrote in Science concerning the 1995 TIMSS assessment, which tested high-schoolers. “In contrast, students in several countries, including the United States, attended comprehensive secondary schools. The major differences in student selectivity and school specialization across countries make it virtually impossible to interpret the rankings.” In TIMSS, especially, tests are conducted by grade-level rather than by age. In elementary and middle school, where topics are often covered and learned over the course of a few weeks, the risk of comparing students at incommensurate stages of their education is great.

Broad curricular differences have probably had a role in deflating U.S. scores. TIMSS and PISA use the same test in every participating country, and the material that makes it onto the test is selected through a winnowing process that leaves the tests considerably narrower than any single country’s general curriculum. Countries that include large amounts of material in their typical curricula are therefore at a disadvantage compared to those countries that focus their curricula more intensely on fewer subject areas. Regardless of its other merits or failings, the American strategy of repeated exposure to a broad range of subjects — American textbooks are the bulkiest in the world — is likely to lend itself to unduly poor performance on standardized tests, as full understanding of any single concept takes longer to develop.

Demographics and culture are also thought to confound the results of cross-national comparisons. In the United States, the tested students come from every socioeconomic rung, while other countries sometimes lack some rungs because of cross-border employment. For example, much of the labor force in Hong Kong (which is treated on the tests as an independent entity) is made up of tens of thousands of low-paid Filipino household workers whose children live and are educated in the Philippines; in light of the extensive literature tying socioeconomic indicators to educational achievement, this cross-border employment surely affects both countries’ scores. A similar situation obtains in other places with significant immigration and cross-border commerce, as Gerald Bracey points out in a 1997 article in the journal Educational Researcher. “Each morning thousands of Malaysians enter Singapore to sweep streets, pick up garbage, and do other low-level jobs. They return to their homes at night, relieving Singapore of having to educate the children of poor laborers,” Bracey writes. “If one reads the [domestic] educational research literature, one is struck by the lengths — the extreme lengths — that researchers go to to ensure that samples in their studies are comparable….The research community would never accept test results in this country that simply compared scores in an inner-city slum and an affluent suburb as if they were comparable,” he writes. The opposite circumstance holds in the United States: Students from all socioeconomic rungs are educated and scored on these tests.

Amid this deluge of confounding factors, the inference that the U.S. education system is going down the tubes is an unjustified logical leap. The United States is still pumping out tremendous numbers of new Ph.D.s in the sciences — more, in fact, than our economy can presently absorb, as there is a well-reported dearth of jobs for newly-minted science Ph.D.s. The same is true in engineering: According to a recent National Science Foundation report, the number of engineers graduating from U.S. schools will continue to grow into the foreseeable future, outstripping the number of available jobs. Of these new engineers and Ph.D.s, an increasing number are foreign-born — but increasing even faster is the percentage of those who decide to stay in the United States. Federal research funding for scientific research and development has consistently risen in absolute terms and as a fraction of discretionary spending — and industry research dollars have risen dramatically on top of that, to the tune of 7 percent per year in real terms — according to calculations by the Consortium for Science, Policy and Outcomes at Arizona State University. (Alarmist media reports often use GDP, against which research spending has fallen, as a comparative baseline.) And countries that have “outperformed” the United States in educational studies for many years — a number of European countries top this list — still fail to rival the U.S. in any measure of research productivity. When Bill Gates and others seem to appeal for school reform in the U.S., perhaps they are merely providing their companies with political cover and a post hoc justification for employing foreign engineers who, while not better educated than U.S. workers, are often significantly cheaper.

Nevertheless, there remains good reason to worry about what the global economy portends for those American students who really are badly educated. In only one other OECD country (New Zealand) are internal educational inequalities worse than in the United States, according to a recent analysis by researchers in England and Italy. Where these inequalities lie is no mystery. The gap in test scores between white and ethnically Asian students on the one hand and black and Hispanic students on the other is a well-known attribute of U.S. schools and is noted ruefully in nearly all cross-national studies. Two University of Pennsylvania researchers recently aggregated scores from a number of cross-national studies and found that white students in the United States, taken alone, consistently outperform the predominantly white student populations of several other leading industrial nations. “There is compelling evidence,” they write,” that the low scores of [black and Hispanic students] were major factors in reducing the comparative standing of the U.S. in international surveys of achievement. If these minority students were to perform at the same level as white students, the U.S….would lead the Western G5 nations in mathematics and science, though it would still trail Japan.” In PISA, for instance, white students performed above most European countries, whereas black students performed on par with students in Thailand. So while the performance of minority groups in the U.S. does refute the alarmist assertion regarding an across-the-board decline in U.S. schools, it does so in a particularly unfortunate way — namely, it suggests that some American minority groups will be shut out of high-paying jobs as companies look for better-educated workers overseas. Although the most recent TIMSS saw the white-black score gap close slightly, it is almost certain to remain shockingly large in the near future.

None of this is to say that other countries are not catching up technologically, nor that the United States is safe from competition in even a single technological sector. China is without doubt the most aggressive challenger. In the mid-twentieth century, Japan’s economy grew 55-fold over the course of thirty years through stringent government control; observers of Japan’s rise will remember the key role of its Ministry of International Trade and Industry, which employed many of the nation’s brightest stars and guided the economy on a carefully directed path of technological growth. China’s strategy has been similar, though its tremendous size has necessitated delegation of heavy-handed economic control to regional governments in what scholars have termed “local state corporatism.” It has simultaneously harnessed the power of markets in a way Japan did not. Regional governments lavish tax breaks on high-tech industries (many of them funded from overseas) and pump millions into China’s new universities — which are poised to graduate more Ph.D.s than the United States by 2010, according to some projections. Nearly all of China’s top leaders are scientists and engineers by training: President Hu Jintao is a hydroelectric engineer, Premier Wen Jiabao is a geological engineer. Their predecessors, Jiang Zemin and Zhu Rongji, were both electrical engineers. The technocrats steering China’s ship of state are working hard to modernize scientific education in their country.

But the United States need not worry — not yet. The U.S. is by no means in technological decline, though China and India will inevitably pose challenges in years to come. Although not a crisis, this competition should motivate the U.S. to improve its science and math education, especially for poor and minority students who might lose out in a globalized, high-tech economy. If sensationalists must take up a cause, it should be the plight of those students and not a hyped-up “threat” of China or the “impending decline” of technological innovation here at home.