About a week ago, Nick Kristof published this op-ed in the New York Times. Titled “Are You Smarter Than an 8th Grader?,” the piece discusses American kids’ underperformance in math compared with students from other countries, as measured by standardized test results. Kristof goes over several questions from the 2011 Trends in International Mathematics and Science Study, or TIMSS, test administered to eighth-graders and highlights how American students did worse than students from Iran, Indonesia, Ghana, Palestine, Turkey, and Armenia, as well as those from traditional high performers like Singapore. “We know Johnny can’t read,” says Kristof, in that finger-wagging way perfected by the current cohort of New York Times op-ed columnists; “it appears that Johnny is even worse at counting.”
The trouble with this narrative is that it’s utterly, demonstrably false.
My friend Jordan Ellenberg pointed me to this blog post, which highlights the problem. In spite of Kristof’s alarmism, it turns out that American eighth-graders actually did quite well on the 2011 TIMSS. You can see the complete results here. Out of 42 countries tested, the United States placed ninth. If you look at the scores by country, you’ll see a large gap between the top five (South Korea, Singapore, Taiwan, Hong Kong, and Japan) and everyone else. After that gap comes Russia, in sixth place, then another gap, then a group of nine closely bunched countries: Israel, Finland, the United States, England, Hungary, Australia, Slovenia, Lithuania, and Italy. Those make up, more or less, the top third of all the countries that took the test. Our performance isn’t mind-blowing, but it’s not terrible either. So what the hell is Kristof talking about?
You’ll find the answer here, in a list of 88 publicly released questions from the test. (Not all questions were published, but this appears to be a representative sample.) For each question, a performance breakdown by country is given. When I went through the questions, I found that the United States placed in the top third (top 14 out of 42 countries) on 45 of them, the middle third on 39, and the bottom third on four. This seems typical of the kind of variance usually seen on standardized tests. U.S. kids did particularly well on statistics, data interpretation, and estimation, which have all gotten more emphasis in the math curriculum lately. For example, 80 percent of U.S. eighth-graders answered this question correctly:
Which of these is the best estimate of (7.21 × 3.86) / 10.09?
(A) (7 × 3) / 10 (B) (7 × 4) / 10 (C) (7 × 3) / 11 (D) (7 × 4) / 11
More American kids knew that the correct answer was (B) than Russians, Finns, Japanese, English, or Israelis. Nice job, kids! And let’s give your teachers some credit, too!
But Kristof isn’t willing to do either. He has a narrative of American underperformance in mind, and if the overall test results don’t fit his story, he’ll just go and find some results that do. Thus for the examples in his column, Kristof literally went and picked the two questions out of 88 on which the United States did the worst, and highlighted those in the column. (He gives a third example too, a question in which the U.S. was in the middle of the pack, but the pack did poorly, so the United States’ absolute score looks bad.) Presto! Instead of a story about kids learning stuff and doing decently on a test, we have yet another hysterical screed about Americans “struggling to compete with citizens of other countries.”
Kristof gives no suggestions for what we can actually do better, by the way. But he does offer this helpful advice:
Numeracy isn’t a sign of geekiness, but a basic requirement for intelligent discussions of public policy. Without it, politicians routinely get away with using statistics, as Mark Twain supposedly observed, the way a drunk uses a lamppost: for support rather than illumination.
So do op-ed columnists, apparently.