If you opened up Twitter on Sunday, Nov. 27, and tried to find the footage of Chinese protests against the government’s extreme Zero-COVID policies, something very curious would’ve happened. Typing in the Chinese names of locations would not have yielded any video. Instead, you would have seen a lot of ads for escorts in your area—your area being Beijing, or Shanghai, or whatever Chinese city you searched for. You would find nothing about the protests. The usernames often included the cities, which was a way to make sure that the account showed up high in the results. Though there hasn’t yet been analysis tracing the specific accounts to known government accounts, in the past, spam floods have been attributed to government or government contractors. That would be the most obvious beneficiary here, according to Joseph Menn, who covers cybersecurity for the Washington Post.
What was Twitter doing while these spam accounts were precluding the actual protest footage from being found? Not much, until Menn brought the issue to the company’s attention. Twitter staff has been cut by roughly two-thirds, through a combination of layoffs, resignations, and firings, which means that the humans who should be making judgment calls about what might be foreign interference are either gone or overwhelmed. Eventually, Twitter began to crack down on the spam accounts, but their slow response bodes poorly for their handling of future challenges.
On Friday’s episode of What Next: TBD, I spoke with Joseph Menn about how Elon Musk’s new bare-bones Twitter responds to crises. We’ve just seen one unsettling test case, and it won’t be the last. Our conversation has been edited and condensed for clarity.
Lizzie O’Leary: Before Elon Musk was in charge, how big was the Trust and Safety team at Twitter, and what did it do?
Joseph Menn: It depends how you count. There are all these sort of overlapping things, which was frankly one of the problems with Twitter. There’s something called Trust and Safety, and there’s something else called Platform Health, there’s Security, and then there’s engineers. One of the problems called out by the whistle-blowing earlier this year is that the Trust and Safety teams didn’t have their own designated engineers. So when they came up with something that would help them root out problems faster, like spam or foreign influence operations, they basically had to wheedle and cajole, and try and convince some other team to let them have an engineer for a little bit.
Under Elon Musk, a lot of the content-moderation people are gone. My understanding is that Twitter is trying to automate some of these functions now to have machine learning do some of this detection and filtering. Is that possible?
I’m afraid this is a big red herring. This is something that Musk has been talking about a lot, and in many other areas. Space-age machine learning is coming to save the day: It’s so much more efficient, and it doesn’t complain about working conditions. That’s not actually the way it works. Twitter has those tools already. It does use them. But there are many things where you need human beings. And for something like this especially.
Suspected Chinese government operatives have used spamming before, but it’s been, for example, in Jinjiang, where the Uyghurs are being put into camps. They have broadcast lots of accounts from happy tourists or happy purported-Uyghurs talking about how wonderful things are, so it’s hard to find actual news about what’s going on there. They have used the sort of porn-y, spammy stuff before, tagging individuals that they wanted to drown out. If you searched for news about person X, you would instead see all this escort crap.
But they haven’t seen this particular thing, where it’s tagged to cities and it’s not about an individual person. So somewhere in the chain, there has to be a human being that says, “Whoops. What’s going on here? This is not helpful. We need to set our tools on this to try and filter it out.” That’s something A.I. can’t do.
What does this Chinese incident tell you? Because it might be, as you’re saying, the first incident of this particular kind of thing during the Musk era. But I’m willing to bet it’s not going to be the last.
That’s the scary thing. This is the first known foreign influence operation under Musk, and it was effective. It is highly likely that there are going to be a lot more from different countries, in part because Musk hasn’t really said anything about foreign influence operations. He’s said he’s a free speech absolutist, and he hasn’t said that there’s an exception for government propaganda. China has objected to the “state-sponsored government official” or “state-sponsored media” labels that are put on some of its accounts. We know that various countries are trying to work the ref here to get less restriction on what they’re doing.
Twitter got a lot better at detecting foreign campaigns on the platform after the Russian interference in the 2016 election was exposed. What happened then?
After that came out in early 2017, they beefed up their staff. When they caught a foreign influence operation, either on their own or because of a tip from Facebook, Google, or the government, they’d keep an archive of which accounts have been suspended and what they tweeted. That was a gold mine for researchers to see, and people are very worried that that’s going to go away. So that’s something that Twitter did better than the other companies. They were more transparent about what influence operations they found.
Now that those teams are gutted, what happens? Do authoritarian governments just say, “Aha, we can spread our message here”?
There’s already foreign influence operations every day on there, [from] multiple countries that we can’t catch. That’s especially true in non-English languages, where Twitter is really understaffed, and they just let go a lot of contract moderators that would catch other stuff. I think there’s going to be more volume, and I think less of it’s going to be caught.
How has Twitter been used as a manipulation tool in other countries in the past, and where you do see that as a potential danger going forward?
There’s one overarching pattern, and that is that authoritarians love Twitter. They can command an army of government employees—or in some cases, the military—to use Twitter and to harness tools to amplify it in various ways, varying degrees of sophistication. It’s been a big problem in autocracies in a variety of places, and governments that are tending towards one-party dominance.
India is a really, really big problem. And India’s especially fraught because it is such a large market that basically no tech company wants to walk away from it. There’s kind of a pincers movement there, where you’ve got the government saying, “You need to take down these protest accounts because they’re terrorists, and that’s our law, so hand over identifying information.” But at the same time, there are party loyalists that are flooding the zone with stuff that could be misinformation. And Twitter has cut its staff in the country radically. There’s one report that says they’ve cut 90 percent of the employees in India.
So far, Twitter has pushed back hard on the Indian government, even taking them to court over the summer over these kinds of requests. Do we have any sense about what might happen now?
There’s a lot of stuff that’s happening below the surface. So it is true that Twitter has taken the government to court, and that has won it a lot of support from civil rights groups and folks in India. On the other hand, you may recall that in the whistleblower account from former chief of security Peiter “Mudge” Zatko, he alleged that there is an Indian government spy inside the company with access to pretty much everything. Sometimes the legal arm is different from the policy arm, and the policy arm is different from the actual moderation of accounts, so it is a very fluid, confusing situation. I’ve spoken to a number of activists in India who are very upset and said there has been an increase in slurs and calls for violence recently on Twitter.
It’s worth noting that Elon Musk’s purported free speech absolutism seems much more focused on the U.S., where speaking up against say, the government, is protected.
Elon himself and many others have been saying, “Look, this is simple. This is just free speech. One tech company shouldn’t be exercising judgment on what’s hateful or whatever. As long as it’s legal, it should be broadcast.” And that is a very attractive argument. But there’s an old saying that the press is free to everybody who owns one. And that’s true with Twitter as well—anybody can have an account in most countries. But it favors the big and powerful because if I’m a company with 100,000 employees or a government with 500,000 employees, I can tell them all to do something that will usually drown out the one or two dissidents, critics, activists. Maybe that’s not as visible in the United States, but it sure as heck is in other countries. Three-quarters of Twitter’s users are not in the United States. And people outside are extremely concerned about this dynamic.
Another potential problem might only emerge in a disaster. Musk’s $8 a month verification plan led to spoofed accounts of the National Weather Service. And that kind of thing could be momentarily funny until it’s really not. Disaster management has actually become this sort of thing that often happens on Twitter, whether we’re talking about a hurricane, a manmade disaster, or a terrorist attack. What have people inside the company been telling you about what might happen in that kind of a situation now?
These plans seem to change a lot, but the latest verification plan is that there will be one for government officials. They get one flavor of check that is a different color than celebrity accounts and the people who just pay the $8 or whatever it’s going to be. So it is likely that once the population gets trained on this new thing, and if it rolls out effectively, you’ll be able to see the right kind of check mark for a county official, or a state, or federal earthquake information.
It is true that many government agencies, especially smaller ones—county governments, school boards, that sort of thing—are looking for different outlets. Sometimes they would just put information on Twitter, but now there’s a Facebook page that they’re going to be posting more on. Or they’re trying to retrain people to search for the actual county government website.
I think a lot of people, myself included, have become used to thinking of Twitter as a place to get news or as a first stop in a crisis, whether that’s from journalists, government accounts, or just people—the guy who basically live-tweeted the Bin Laden raid. How do you think that’s going to change? Is it?
If people leave the site, then yeah, over time it’ll be less of a platform for citizen reporting or whatever you want to call it. And something will have been lost there. I think that will happen if it’s all vitriolic. At the moment, we know that some slurs are up. We know that there’s a lot of discussion on Twitter about Twitter, which isn’t really terribly interesting to most folks. It’ll make it less attractive over time. People said in the early days of this that there aren’t really any good alternatives to Twitter, and I think that was true. But it has been surprising, those alternatives that are out there, how quickly they’ve been attracting people. Mastodon is one, Hive is another.
I think there’s a real chance that we won’t have this one giant public square anymore, where you can go for anything. I think things may be more specialized, because it turns out that if you have a giant, basically unregulated public square, then there’s a lot of propaganda and noise and commercial stuff, which is not really very interesting or useful to people.
If we can’t rely on Twitter, and I don’t know yet if we can rely on those other alternatives—to combat propaganda, foreign interference, or get the message out about an earthquake—does that leave people worse off? Are there other places that can fill this gap, or is there going to be something missing?
Having written about this stuff for 20 years, one of the surprises is that things that people think are irreplaceable tend to be replaceable. People thought that MySpace could never be beaten by an upstart, and then it happens. Once there’s a tipping point, and everybody goes to something else. And I think there’s a reasonable chance that we’re at one of those, that Twitter as we know it is going to wither away, and that people will wind up at one big new place.