Madea’s Family Reunion took first place at the weekend box office, racking up $30,250,000. No one sat down and counted the money, of course. The seemingly meticulous numbers are just estimates. Likewise for Lost’s Nielsen ratings, Howard Stern’s Arbitron scores, and Beyoncé’s No. 1 spot on Billboard’s Hot 100. All these are educated guesses, but executives believe in the figures enough to hire and fire over them. Fox canceled Arrested Development because of its low Nielsens, while Madea’s Family Reunion 2 has probably been greenlighted already.
There’s only one media-ratings system no one takes seriously: Web site rankings. You never see Harvey Weinstein on Entertainment Tonight telling viewers not to believe the latest box-office tallies. But confront any Internet mogul with the numbers published by the two big Web ratings houses, Nielsen/NetRatings and comScore Media Metrix, and you’re in for an earful about how they’re wrong, wrong, wrong.
Web publishers are so confident about this because they have their own numbers. A site’s internal log files track every single page served, and to where. When publishers compare these internal counts against third-party ratings, they usually conclude that Nielsen and comScore’s official rankings are too low. If nothing else, the two companies are often way out of sync. My fellow Slate scribe Adam L. Penenberg pondered this “fuzzy math” last year. Nielsen measured Wired News’ audience at 1.096 million, while comScore reported an equally precise-sounding 1.87 million. Which was right, Penenberg wondered—1 million, 2 million, or neither?
It’s obvious why nobody trusts site rankings—the “official” numbers from Nielsen and comScore are often so far apart that you couldn’t trust them even if you wanted to. But site owners have a real stake in these numbers. Advertisers use them to decide where to spend their budgets, and the inconsistent counts make buyers wary. So, if Web moguls think the published statistics are inaccurate, why don’t they just release their own numbers? They will—if you’re a prospective advertiser on the phone. Otherwise, they prefer to keep house stats under wraps because they see them as competitive info.
Slate’s publisher Cliff Sloan has given me permission to share the numbers from our internal traffic report for December. (Slate licenses software from IBM that compiles monthly, weekly, and daily traffic reports.) The chart below, which compares Slate’s numbers to those from Nielsen and comScore, shows two different metrics for looking at site traffic:
Page views are the number of individual Web pages the site served. If one person reads a three-page story, that’s three page views. If you reload or revisit the page later, that’s another page view. Publishers use this number to measure available inventory for advertisers, who typically pay based on how many times their ads will be shown.
Unique visitors are the number of people who looked at the site. If you look at the Slatehome page at home and at work, that counts as two page views but only one unique visitor. Publishers use this number to give advertisers an idea of the site’s “reach”—that is, how many people might see an ad.
Nielsen and comScore agree on Slate’s audience for the month, but the rest of the measurements are far apart.
|Nielsen//NetRatings||comScore Media Metrix||Slate internal report|
|Unique Visitors||4,600,000||4,609,000||8,051,709 unique browsers|
Why the different numbers? For one thing, Slate’s 8 million browsers came from all over the world. Nielsen and comScore count only U.S. visitors, the ones most of our advertisers are trying to reach.
While Slate’s stats come straight from the server logs, Nielsen and comScore employ panelists who get modest compensation to install tracking software on their computers. The companies use different formulas to extrapolate total traffic from their panel measurements.
There are problems with both methodologies. Notice that Slate counts unique browsers instead of unique visitors. It’s easy to count browsers by tagging each one with a cookie. But if the same person uses one browser at home and another at work, that will cause an overcount. You get an undercount if the same browser is used by more than one person. You can only guess roughly how many surfers have deleted or disabled their cookies, which can scramble the count both ways. And some unknown number of browsers aren’t people, but robots crawling sites to index them for search engines or price-comparison sites. (Google identifies itself, but some content-scrapers deliberately camouflage themselves to avoid being blocked.)
Altogether these factors usually add up to massive overcounting—in this case, 8 million browsers instead of 4 million unique visitors. Web publishers swap notes with each other and apply their own math to adjust for multibrowser users and cookie haters, but the result is an approximation rather than a straight count. Slate subscribes to NetRatings’ advanced client reports for a second opinion.
The ratings companies, for their part, can’t accurately measure people surfing from universities, government agencies, and large corporations. That’s because many such places prohibit workers from installing the tracking software. Academics, government workers, and corporate professionals—that sounds like a good chunk of Slate’s core readership. This undercounting makes for a good argument that Nielsen and comScore are undercounting sites like Slate. (Nielsen tries to even out the work-surfing numbers by conducting random-dial phone surveys, while comScore solicits a larger panel—120,000 to Nielsen’s 28,000.)
The two agencies agree on the number of Slate’s December visitors, but the results weren’t so cozy for the top news sites. (To compare the Nielsen and comScore ratings for the Net’s top five news sites, click here.)
As for page views, it’s common to hear publishers say—off the record, of course— that they have twice the traffic estimated by Nielsen. CNN and Fox execs recently made such claims to the New York Observer. In this case, you should believe the publishers. Rather than extrapolating from a sample, they log every single page request made to their servers and subtract the ones they don’t think are valid (like robots or their own staff). They might adjust the count up for subscribers who read by e-mail, and adjust down for slide shows that rack up a dozen page-views for one article. They use the results to decide what topics are hot, which writers bring in a lot of readers, and whether slide shows are worth the effort. It’s in the publishers’ interest to get their page-view counts right, and they’ve got the server logs to do it.
If you’re not an ad buyer, you probably only care how many readers a site has. That requires a unique-visitor count, where it’s in the publisher’s interest to exaggerate their reach to sway potential advertisers. Nielsen and comScore don’t offer NASA-grade precision, but they’re the most studiously calculated numbers available and the most consistent means to compare one site against another. How do you reconcile the factor-of-two difference between their Wired News counts? You accept the awkward truth: We aren’t sure of the right answer, but we’re closing in on it.
Penenberg called this “flying blind.” I take the opposite view: It’s transparency, one of the Net’s basic strengths. Conflicting traffic reports confuse ad buyers, but they also give them more clues about where their money is really going. Mismatched ratings drive Web publishers crazy, but they can (and do) call Nielsen and comScore on the carpet to reconcile their methods against internal counts. I’m sure Harvey Weinstein can put the screws to the box-office people when he doesn’t believe them, but he can’t hit them with 25 terabytes of server logs.
The more I dig into how Web ratings work, the more I realize people in other media are in denial. Internet publishing is the most finely measurable medium ever invented; broadcast, movie, and print companies have no way of monitoring individual transactions from their end. Yet, while the Web guys admit they could be off by half, Nielsen claims its television ratings have a margin of error of 4 percent. If I were in the cast of Arrested Development, I’d demand a recount.