Twitter’s Vulnerabilities, Exposed
Joseph Menn: Well, don’t go back there. Hey.
Lizzie O’Leary: If you opened up Twitter on Sunday and tried to find this footage, footage of Chinese protests against the government’s extreme zero COVID policies, something very curious would have happened. Typing in the Chinese names of locations would not have yielded any video.
Joseph Menn: You’d see a lot of ads for escorts in your area, your area being Beijing or Shanghai or whatever city you were looking for. And you would find nothing about the protests.
Lizzie O’Leary: That’s Joseph Menn, a veteran tech reporter for The Washington Post.
Joseph Menn: And sometimes, like the users, had the the names of the cities. So it was just, you know, the person was named Shanghai or something like that. And this was a way to make sure that it showed up high in the listings. But there were just hundreds and hundreds of these. Some other adult stuff, too. But it’s mostly very repetitive, like the same few over and over again.
Lizzie O’Leary: What do we know about who created them?
Joseph Menn: So this falls under the heading of reasonable speculation. You know who who benefits? I haven’t seen analysis to trace the specific accounts to known known government accounts. There are some that had been dormant for years. There are some that had, you know, just tweeted a few times. But, you know, not not in several months. And then there were some, you know, a whole bunch of new ones. In the past, there have been spam floods that have been attributed to government or government contractors. And that would be the most obvious beneficiary here.
Lizzie O’Leary: And what was it like inside Twitter when this was happening?
Joseph Menn: Well, that’s the thing. Twitter is a place of chaos right now. The staff has been cut by roughly two thirds, either through layoffs or resignations and or some firings. And the people who are left are really outgunned and they’re especially outgunned in sort of policy policy areas and trust and safety.
Lizzie O’Leary: That means the humans making a judgment call about what might be foreign interference are either gone or overwhelmed.
Joseph Menn: That was badly depleted and foreign language specialists were badly depleted. So they actually they were slow to pick up on this happening. And actually, after after I, I got in touch with them, they started to crack down on this and it got it got better.
Lizzie O’Leary: Today on the show, how will Twitter, Elon Musk’s new barebones Twitter respond to crises? We’ve just seen one unsettling test case. It won’t be the last. I’m Lizzie O’Leary and you’re listening to What Next TBD a show about technology, power and how the future will be determined. Stick around. I want to back up a little bit and get a sense of what things were like before Musk took over Twitter. So before Elon Musk was in charge. How big was the trust and safety team at Twitter and what did it do?
Joseph Menn: It depends how you count. You know, there are all these sort of overlapping things, which was frankly, one of the problems with Twitter. There’s there’s something called trust and safety and there’s something else called platform health. And then there’s security and then there’s engineers. And one of the problems sort of called out by the whistleblowing earlier this year is that the teams, the trust and safety teams didn’t have their own designated engineers. So when they they came up with something that would help them root out problems faster, like spam or foreign influence operations or whatever, they basically had to wheedle and cajole and try and convince some other team to let them have an engineer for a little bit. So that wasn’t it wasn’t terrifically effective before.
Joseph Menn: So trust and safety includes stuff like harassment, abuse, you know, brig, what’s called brigade thing where like a whole bunch of people show up and in somebodies mentions and harass them, doxing, you know, all these sort of bad things that happen. And it’s mostly, you know, it’s mostly sort of individualized, the fancier stuff where it’s like nation state activity. We’re not talking about huge numbers and couple of dozen, that sort of thing. And so they you know, they worked hard and they got assistance from other parts of the company, but they sort of labored on their own, you know, on their own. And now those guys are down to very, very few.
Lizzie O’Leary: Let’s say Twitter was still a public company and this Chinese influence operation happened. What would things have looked like inside the company then? Is there any way to know?
Joseph Menn: Not precisely, But I spoke to people inside the company and they and they said that they would have done better before. It’s a matter of resources and and’s and specialties and, you know, institutional knowledge and all that. There are these tools. But even before it was sort of cobbled together, it was like it was jerry rigged. Like you had to you had to use multiple systems.
Joseph Menn: For example, if if somebody sent you a piece of propaganda and then the next day under one account, and then they changed like one word and sent it over over a second account, there wasn’t actually no automated process to catch that. The people inside Twitter would have to main you catch that and then manually enter it into, okay, wait, block this thing too. And so, you know, there are other parts of Twitter that are very technically sophisticated, but not this part. So the people inside Twitter told me they would have been able to catch this and put it out faster. They’re just you know, the fewer the there would have been more people to throw at the problem. And and they would have they would have been able to get it reasonable again.
Lizzie O’Leary: Under Elon Musk, a lot of those people are gone in their place. Elon Musk has been promoting machine learning to handle content moderation automatically. But Joel Roth, Twitter’s former head of Trust and safety, said it’s the human element that’s really necessary.
Elon Musk: Twitter’s challenge going forward is not, you know, can the platform build machine learning? Sure they can. But are there enough people who understand the emergent malicious campaigns that happen on the service and understand it well enough to guide product strategy and policy direction? And I don’t think that there are enough people left at the company who can do that work.
Lizzie O’Leary: My understanding is that Twitter is trying to to automate some of these functions now to to kind of have machine learning, do some of this detection and filtering. Is that possible?
Joseph Menn: So I’m afraid this is kind of a big red herring. This is something that the that Musk has been talking about a lot and not just in this area, but many other areas. You know, space age, A.I. machine learning is coming to save the day and and so much more efficient. And it doesn’t complain about working conditions. That’s not actually the way it works. You know, Twitter has those tools already. It does use them, but they’re things where you need human beings. There are many things where you need human beings. I was talking to a human rights person yesterday who’s been active on dealing with Twitter in the past. And they’re and there’s know they said AI’s not coming to save the day. You need people to make judgment calls and and something like this, especially where you haven’t seen this particular technique before.
Joseph Menn: So suspected Chinese operatives, government operatives have used spamming before, but it’s been, for example, in Xinjiang, where there’s the wiggers are being put into camps and there’s there’s you know, there’s there’s great upset. They have broadcast, you know, lots of accounts from happy tourists or, you know, happy purported wiggers, you know, talking about how wonderful things are. And so it’s hard to find actual news about what’s going on there. And they have used the sort of Pawnee spammy stuff before tagging individuals that they wanted to drown out.
Joseph Menn: So if you searched for news about, you know, person X, you would instead see all this, you know, escort crap. But they haven’t seen this particular thing where it’s tagged to cities and it’s not about an injury person or whatever. So a somewhere in the chain, there has to be a human being that says, Woops, what’s going on here? This is not helpful. We need to like, you know, set our tools on this to try and try and filter down. And that’s something that I can’t do.
Lizzie O’Leary: Yeah, I noticed some some reporting from NPR this week that Twitter software failed to to detect some videos from the Christchurch, New Zealand attack that the people we posted those and Twitter didn’t catch it.
Joseph Menn: Right. So apparently that has happened before, but it doesn’t last as long. I mean, it’s very fact specific. You know, you have to look at each incident and yes, they’re signatures and yes, they’ve been able to catch it before. It’s not tracking something that something to me would go up for like a minute or two. But any widespread circulation should have been stopped.
Lizzie O’Leary: What does this Chinese incident tell you? Because, you know, it might be, as you’re saying, the first incident of this particular kind of thing during the Musk era. But I’m willing to bet it’s not going to be the last.
Joseph Menn: No, that’s the scary thing. This is the first known sort of foreign influence operation under under Musk, where they had they had any difficulty responding and it was effective. It is highly likely that there are going to be a lot more from a lot more different countries, in part because Musk hasn’t really said anything about foreign influence operations. You know, he said he’s a free speech absolutist and he hasn’t said that there’s, you know, an exception for for government propaganda. So we don’t know. You know, China has objected to the state sponsor, you know, government official or state sponsored media labels that are put on some of its accounts. We know that the various countries are trying to work the ref here and try to try to get less restriction on what they’re doing.
Lizzie O’Leary: Joseph says Twitter got a lot better at detecting foreign campaigns on the platform after the Russian interference in the 2016 election was exposed.
Joseph Menn: So after that came out in early 2017, they beefed up their staff and among other things, when they caught a foreign influence operation either on their own or because of a tip from Facebook or Google or the government, they would keep it. Archive of which accounts have been suspended and what they tweeted. And that was a gold mine for researchers to see. Okay, what are the strategies? Who else are these accounts connected to that that Twitter didn’t suspend but are sort of part of a broader cloud of activity for one reason or another, either because they’re, you know, ideological allies or there was money changing hands or whatever. And it’s been amazing. And people worried are very worried that that’s going to go away. So that’s something that Twitter did better than the other companies. They were more transparent about what influence operations it found.
Lizzie O’Leary: You know, just a few weeks ago, in fact, they were talking Yoel Roth, who used to be there, was was showing some of this research. And I wonder now that those teams are gutted, what happens? Do do authoritarian governments just say, aha, we can we can spread our message here?
Joseph Menn: So they’ve already been doing that. It’s just the bigger ones get caught, the more obvious ones get caught. And so I don’t think it’s going to you know, there’s still some people left. I think they’ll still catch some. But I think the bar is going to be much higher for getting detected. So you could have a you know, I mean, there’s there’s already foreign influence operations, you know, every day on there, multiple countries that we can’t catch. That’s especially true in other non-English languages where Twitter is is really understaffed and they just let go a lot of contract moderators that would catch other stuff. I think there’s going to be more volume and I think less of it is going to be caught. At the same time, it’s just going to be like the really big blatant ones.
Lizzie O’Leary: When we come back.
Lizzie O’Leary: If there’s an earthquake, should you still trust what you read on Twitter? I think one of the things that’s been happening in the U.S., in U.S. media is, you know, a lot of conversation centered on Elon Musk, understandably. But I’m really curious how this affects kind of global safety. Global manipulation. I’d love to know from you kind of how Twitter has been used as a manipulation tool in other countries in the past. And and where you see that as, you know, a potential danger going forward.
Joseph Menn: Well, there’s one sort of overarching pattern, and that is that authoritarians love Twitter. People that can command an army of government employees or in some cases like the military, to use Twitter and and to, you know, harness tools to amplify it in various ways of varying degrees of sophistication. They love it. So it’s been it’s been a big problem in autocracies in a variety of places and governments that are tending towards, you know, one party dominance. Brazil has been really bad. There’s just been a change of of power or there’s about to be a change in power in Brazil, and maybe it’ll get less bad. But there are also all these people with skills on how to manipulate Twitter who are going to be looking for new employment. So they might just, you know, be doing it for somebody else now.
Joseph Menn: India is a really, really big problem. And India is especially fraught because it is such a large market that basically no no tech company wants to walk away from it. So one of the dynamics you have there is that there’s kind of a there’s kind of a conservative movement there where you’ve got the government saying you need to take down these protest accounts because they’re terrorists and that’s our law. So hand it, you know, hand over identifying information, etc.. Suppress this thing, suppress that thing, label the other thing. But at the same time, there are party loyalists that are flooding the zone with stuff that could be misinformation. And Twitter has cut its staff in the country radically. There’s there’s one report that says they’ve cut 90% of the employees in India.
Lizzie O’Leary: So far, Twitter has pushed back hard on the Indian government. You know, took them to court over the summer over these kinds of requests. Do we have any sense about what might happen now?
Joseph Menn: There’s a lot of stuff that’s happening below the surface. So it is true that Twitter has taken the government to court and that has won it. You know, a lot of a lot of support from civil rights groups and folks in India. On the other hand, you may recall that in the whistleblower account from former chief of security, Peter Mudge Zenko, he alleged that there was an Indian government spy inside, inside the company with access to pretty much everything. So, you know, you know, sometimes the legal arm is different from the policy arm, and the policy arm is different from, you know, the the the actual moderation of accounts. So it is a very fluid, confusing situation. And I’ve spoken to a number of activists in India who are very upset and said there has been an increase in and in slurs and violence calls for violence recently on Twitter.
Lizzie O’Leary: It’s worth noting that Elon Musk’s purported free speech absolutism seems much more focused on the U.S., where speaking up against, say, the government is protected.
Joseph Menn: Elon himself and many of his many others have been saying, look, this is simple. This is just free speech. Everybody, you know that one tech company shouldn’t be exercising judgment on what’s hateful or whatever. You know, as long as it’s legal, it should be broadcast. And that is that is a very attractive argument. But if everybody can talk, there’s an old saying that the press is free to everybody who owns one. And that’s that’s true of Twitter as well. Anybody can have an account in most countries.
Joseph Menn: But it favors it favors the big and powerful. Because if I can have, you know, if I’m a company with 100,000 employees or a government with, you know, 500,000 employees, and I can all tell them to do something that will usually drown out, you know, the one or two dissidents, critics, activists types. So that is that is you know, maybe that’s not as visible in the United States. But it sure as heck is in other countries. And three quarters of Twitter’s users are not in the United States. And the people outside are extremely concerned about this dynamic.
Lizzie O’Leary: Do you think Elon Musk thinks about. That part at all, because at least as far as what he’s saying publicly, he really seems to be focused on the U.S..
Joseph Menn: So I don’t like trying to analyze what’s in a person’s head. You know, I can only judge what what I see and what other people tell me. I have not heard him say very much about things in other countries. I guess the exceptions have been when either European regulators say, well, look, we have some new laws coming online and you are going to comply. And that includes, you know, having a system for attacking misinformation, you know. You know, he takes those calls. The only other time he really talks about the countries that I see is when he is talking about, you know, how Tesla, which was the major source of his money, is going great guns in China or, you know, or something like that. He has a lot of business interests in other countries. And this is a source of of concern in the US government.
Lizzie O’Leary: One thing that has certainly been a source of concern for human rights activists and others who have been kind of watching Twitter’s dealings in other countries is the size of the Saudi stake in this new iteration of Twitter. I wonder if people communicated any concerns to you about that.
Joseph Menn: Yeah, sure. I mean, Saudi Arabia and other countries have a stake of less than 10% in in in Twitter. And to be fair, Saudi Arabia had a stake when it was a public.
Lizzie O’Leary: Company that just rolled it over.
Joseph Menn: That’s right. On the other hand, one of my colleagues found out that if you have a stake of more than $250 million, you have certain extra rights in the new company you get you get information that others do not. And that is that is a source of concern, because even just information about what Twitter’s plans are is is is useful to to one government or another. And, you know, if it comes with any chance to influence the decision making, then that that that is more of a problem.
Lizzie O’Leary: Another potential problem might only emerge in a disaster. Already we’ve seen Musk’s $8 verification plan lead to spoofed accounts of the National Weather Service, and that kind of thing could be momentarily funny until it’s really not. Disaster management has actually become this sort of thing that often happens on Twitter, whether we’re talking about a hurricane or, you know, a manmade disaster or a terrorist attack. What have people inside the company been telling you about what might happen in that kind of a situation now?
Joseph Menn: Well, so a couple of things. First of all, the latest and these these plans seem to change a lot, but the latest verification plan is that there will be one for government officials. They get one flavor of check that is a different color than, you know, celebrity accounts. And and then the people who just, you know, pay the $8 or whatever it’s going to be. So it is likely that you will still be able to find once you once the population gets trained on this new thing. And if it sticks and if it rolls out effectively, you will be able to see, you know, you know, if you’re looking for earthquake information from some city in California, you’ll be able to see the right kind of check mark for, you know, a county official or state or federal earthquake information. And you’ll still be able to get things that way.
Joseph Menn: I think the transition is going to be awkward. It is true that many government agencies, especially smaller ones, county government school boards, that sort of thing, are looking for different outlets. Sometimes they were just, you know, they would just put information on Twitter. But now there’s a Facebook page that they’re going to be posting more on or they’re trying to retrain people to, you know, search for the actual county government website, that sort of thing.
Lizzie O’Leary: I think a lot of people, myself included, have become used to thinking of Twitter as a place to get news or be your first stop in a crisis, you know, whether that’s from journalists or government accounts or just people. I mean, the guy who basically live tweeted the bin Laden raid. How do you think that’s going to change? Is it?
Joseph Menn: That’s a really interesting question. If if people leave the site, people stop spending as much time on it and they check in, you know, once a day or once a week or something, then, yeah, over time it’ll it’ll be less of a platform for, you know, citizen reporting or whatever you want to call it, and something will have been lost there. And I think that will happen if it’s all vitriolic. I mean, at the moment, you know, anecdotally, we know that some slurs are up. We know there’s a lot of discussion on Twitter about Twitter, which isn’t really terribly interesting, I think, to most folks. And it will make it less attractive over time.
Joseph Menn: People said in the early days of this a month ago that there aren’t really good alternatives to Twitter. And I think that was true. But it has been surprising, you know, the alternatives that are out there, how quickly they’ve been attracting people. Mastodon is one, hive is another. And, you know, I think there’s a real chance that we won’t have this one giant public square anymore where you can go for anything.
Joseph Menn: I think things may be more specialized because it turns out that if you have a giant, basically unregulated public square, then there’s a lot of propaganda and noise and commercial stuff, which is not really very interesting or useful to people. You know, maybe it’ll still be the place for celebrity news, but even some of those or who have left back when they’re on Instagram or whatever. So I think there’s a real chance that you will find your community. I mean, one of the great things about Twitter is the sub communities there, you know, black Twitter, golf, Twitter, whatever. And I think maybe those groups will go to different places, you know, one or another server on Mastodon or something else.
Lizzie O’Leary: If we can’t rely on Twitter and I don’t know yet if we can rely on those other alternatives to to combat, say, propaganda, foreign interference or, you know, get the message out about an earthquake, does that does that leave people worse off? Like, are there other places that can fill this gap or is there is there going to be something missing?
Joseph Menn: Well, having written about this stuff for 20 years, one of the surprises is that things that people think are irreplaceable tend to be replaceable. People thought, you know, the MySpace could never be beaten by an upstart. And then it happens. You know, once it there’s a like a tipping point and everybody goes to something else. And I think there is a there is a reasonable chance that we’re at one of those that Twitter as we know it, is going to wither away and that people will find, you know, will, you know, wind up at one big new place.
Joseph Menn: You know, I said that maybe there will be lots of different places, but there could also be another big place because the network effects you know, what that means is that the value of a network goes up proportionally with every new node, with every new member. And, you know, you want to go to a place where all your your friends are the people with the resources you care about are. And once there’s a, you know, it starts growing, then more people check it out and then that makes them more valuable and more people grow. And it could actually happen fairly quickly. So that has happened over and over again with social networks. And there’s no reason to think it can’t happen again.
Lizzie O’Leary: Joseph Menn, as always, it’s a delight to talk to you.
Joseph Menn: Thanks so much.
Lizzie O’Leary: Joseph Menn covers cyber security for The Washington Post. And that is it for our show today. What next? TBD is produced by Evan Campbell. Our show is edited by Jonathan Fisher. Joanne Levine is the executive producer for What next? Alicia montgomery is vice president of Audio for Slate. TBD is part of the larger What Next Family and were also part of Future Tense, a partnership of Slate, Arizona State University and New America. And if you’re a fan of the show, I have a little request for you. Become a Slate Plus member. You get all your podcasts ad free. Just head on over to Slate.com slash what next? Plus, to sign up. It makes a lovely holiday gift. We will be back on Sunday with another episode. I’m Lizzie O’Leary. Thanks for listening.
