Game Google, Help the World

Why search-engine optimization makes the Web a better place.

Illustration by Rob Donnelly. Click image to expand.

The dating service Together bills itself as one of those high-end matchmakers that still connect people the old-fashioned way, face to face. Though they’ve been around since 1974, the company has long since expanded onto the Web. Google “together dating,” and their site is the first result.

Unfortunately for Together, the next two results that Google delivers are from a site called Ripoff Report, which allows people to air grievances anonymously. One of the results links to a complaint from a man named Gary in Crystal, Minn., who bought a contract with Together off a friend for $2,300. Says Gary:

They admit there are not many girls the size I desire 135lbs or under. Being that I’m 165lbs I like to be a little bigger then my date. So to sum it up in 3 ½ years I had 5 dates for $2.300.00. And my friend before me had 4 dates for a $1.000.00 for the grand total of $3.300.00. What a deal!

The other result links to all 51 complaints about Together that people have filed with Ripoff Report.

That almost certainly hurts Together’s business; favorable or at least neutral search results are essential for any firm with a presence online. While the most compelling stories from the 650,000 search queries that AOL released in August 2006 were the bizarre things people type into search engines, the data also had a lot to say about the importance of ranking at the very top of the results. As the Guardian reported, the first result got 42 percent of all click-throughs, while the second got 11 percent and down from there.

Companies will do whatever it takes to stay at the top—and ahead of their critics. To that end, a variety of firms promise to drown out the bad publicity with press releases and other friendly content. One even specifically promises to drown out Ripoff Report with “satisfactory rankings from the opinions of Bloggers throughout the Internet.”

At first, this practice might offend our sense that the Web should empower the little guy who’s up against the big bad corporation. I’ve argued before that democracy in social media is largely a myth. In this case, I’m not convinced that the practice of optimizing press releases for prime search position is such a terrible thing. In fact, the competition it fuels should make the Web a better place.

Gaming search engines has gotten a lot more sophisticated since the days of flooding your site with the word sex in invisible text. “Search-engine optimization” is now an enormous industry teeming with both legitimate business—those who simply want to help Google help you—and shadier consultants who promise that, by hell or high water, they’ll get you to the top of the results—at least for a few days. In the Web idiom, they are referred to as white hats and black hats, and the former crowd usually doesn’t appreciate getting lumped in the same industry as the latter. (Here’s a decent rundown of white-hat vs. black-hat techniques.) As usual, there are lots of gray fedoras in between.

Bear in mind that the Internet is a hugely disorganized, dysfunctional mess, rife with sloppy HTML, broken code, and millions of software-generated pages of links that no human would ever want to read. Pity the program that has to make sense of it. Most sites are miserably un-optimized for attention from search engines because many Web masters simply lack know-how to take the basic steps toward making their sites search-friendly. Those who do, or who have the money to hire people who do, are at an enormous advantage.

It’s tempting to approach this subject with the consumer-protection assumption that a customer’s criticism is fundamentally more honest and relevant than a company’s press release. This gets a bit sticky with sites like Ripoff Report or its brethren, like or the Squeaky Wheel. Most allow anonymous posting and do nothing to verify the veracity of their claims. Gary from Crystal, Minn., could just as easily be an employee of, say,

Because these sites aggregate thousands of complaints under one domain, they pack a lot more clout in terms of relevance, at least on some search engines. (Notice that a Yahoo search for together dating churns up a lot less criticism, as does one on Live Search.) It is very unlikely that Gary from Minnesota—assuming he’s real—would have made it to No. 3 on Google’s results if he posted his laments on a personal blog without much readership.

A big site like this can be tough to shake from the top of a ranking. “It’s very easy to change up the mix-up in the second page or the bottom of the first page” of the results, says Andy Ball, the president of USWeb, an SEO firm. Drowning out unflattering material in the two or three spot, on the other hand, “is going to take a long time and a lot of effort.”

This is the fundamental arms race of the Web right now, and it bears a strong resemblance to natural selection; companies with robust sites that play nicely with Google’s spiders—that is, its army of bots who read and index pages—are rewarded with more eyeballs and better odds of survival.

There’s one important caveat: The search engines write the rules, and they can rewrite them to discourage people who find a loophole in the system. They can remove and redeem individual players on demand, counsel the lower organisms in ways to succeed, and ruffle the playing field at will. (The New York Times reported last June that Google makes about a half-dozen changes to its algorithm a week. A Google spokesperson told Slate that the company averaged more than one change a day in 2007.)

If—and this is an important if here—search engines write and maintain the rules to reward good behavior, like entering relevant keywords and metadata that help connect search terms with content—then searching will become a better experience for everyone. So long as sites are rewarded with higher ranking for making their sites more search-friendly, competition will spur better, more relevant results.

As it stands, many companies are still very poorly optimized for search, making it harder for them to get noticed and harder for people who are genuinely interested in their sites to find them.

As companies begin playing SEO hardball, other content providers, like media sites, will have to improve as well, spurring innovation from both parties. It helps Google, too, because it gives them better material to work with when deciding, across a broad spectrum of sites, which ones are more relevant than others—an admittedly vague qualification. In their 1998 research paper that introduced Google, founders Sergey Brin and Larry Page acknowledged that this quality is in the eye of the beholder. As they wrote, the maps that their algorithm created, using hyperlinks from other sites as the fundamental electors of relevance on the Web, provided “an objective measure of its citation importance that corresponds well with people’s subjective idea of importance.”

As companies spend more money paying other sites for a few good links, for example, search engines will have to get smarter about detecting paid links, all with the goal of rewarding Web sites for behaving in ways that make them easier to search and target for specific, relevant pages.

The burden falls to the search engines to build and maintain an environment that can turn this inevitable arms race for rankings into precise search results. Thus far, they have proved to be a decent bet in this task.