A 5-4 Vote Against? Or 6-3 in Favor?

How reliable are we at predicting the outcomes of cases before the Supreme Court?

Jeffrey Toobin.
CNN legal analyst Jeffrey Toobin called the oral arguments on the Affordable Care Act a “train wreck” for the Obama administration and predicted the Supreme Court will strike down the health care reform law.

Photograph by Amy Sussman/Getty Images.

The Supreme Court heard oral arguments this week on the constitutionality of the 2010 health care reform law. The consensus among the media is that things didn’t go well for President Obama, and some reporters are predicting that the individual mandate will be declared unconstitutional. After making their assessments, court observers usually add the caveat that you can’t reliably predict the outcome of a case based on oral argument. Is that old saw true?

No. For years, the conventional wisdom among legal scholars was that oral arguments had little or no impact on the outcome of a case, and that any predictions based on them would be unreliable. Accordingly, Supreme Court forecasting models used to be based only on contextual information, such as the subject matter being decided (tax law, federalism, civil rights, etc.), the relative wealth of the parties, and whether the lower court decision was conservative or liberal. The best of these complex models correctly predicted about 75 percent of cases and performed extremely poorly on nonideological cases involving tax or maritime law. (As a general rule, legal scholars are able to call around 60 percent of cases correctly, simply on the basis of their expertise and intuition.) Then, in 2004, New York Times Supreme Court reporter Linda Greenhouse noticed that she could at least match the top forecasting models simply by listening to the tenor of oral arguments. Greenhouse argued that the justices tip their hands during questioning, so models that ignored this phase of a case were incomplete.

Also in 2004, future Chief Justice John Roberts, then a Supreme Court attorney, noticed that justices ask more questions of the side that ultimately loses in 86 percent of cases. Though he’d tested his theory on only 28 cases—and no one knew whether its sensational success rate would hold up—there was now ample evidence that oral arguments do, in fact, offer very useful data for predicting the outcome of a Supreme Court case. Drawing on this insight, many political scientists now incorporate language analysis of the arguments into their models. One method is to simply tally up the words spoken to each attorney, consistent with the approach Chief Justice Roberts pioneered. The other method goes slightly deeper, analyzing the emotional quality of the language that the justices use: The more “unpleasant” words a lawyer hears compared with his opponent, the less likely he is to win. (Justice Scalia has a habit of telegraphing his vote by using words like “idiotic” during oral argument.) The most sophisticated models combine these two strategies with the more traditional forecasting factors—ideology, historical voting patterns, etc. These haven’t yet been deployed in enough cases to have a documented success rate, but research on past cases strongly suggest a major improvement in accuracy.

Predicting the outcome of the present health care reform case is difficult, because there are several different questions being debated and more than two attorneys appearing before the court. However, a state-of-the-art model created by professors Ryan Black of Michigan State, Sarah Treul of the University of North Carolina, Timothy Johnson of the University of Minnesota, and Jerry Goldman of Chicago-Kent College of Law suggests that the court will declare the individual mandate unconstitutional by a 5-4 vote. The big question mark, of course, is swing voter Justice Kennedy. He asked Solicitor General Donald Verrilli Jr. two more questions than he asked the challenger’s attorney, Paul Clement, with 14 percent more negative language, suggesting a slight preference for overturning the law. 

Got a question about today’s news? Ask the Explainer.

Explainer thanks Timothy Johnson of the University of Minnesota and Jeffrey Segal of Stony Brook University.

Video Explainer: Why Aren’t There More Slam Dunks in Women’s Basketball?

This video was produced from an original Explainer by Brian Palmer. Want more questions answered? You can now watch video Explainers at Slate’s News Channel  on YouTube.