We Checked Elon Musk’s Claims About Twitter Bots; Here’s What We Found
Share
For internet watchers and security researchers, Elon v. Twitter was the story of the summer. Twists, turns, public insults, we got it all. The saga drew to a close in October, when Elon Musk moved forward with his $44 billion buyout of Twitter. In his first e-mail to Twitter staff, Musk reportedly discussed making the battle against bots a priority. Last week, he posted about Twitter actively purging spam accounts. So even though the leadup to Musk acquiring Twitter is in the rearview mirror, certain questions remain: namely, how many bots are there still on Twitter, exactly?
The percentage of bots on the platform was a frequent point of contention which Musk has repeatedly attempted to leverage as a way out of the deal.
To report its user base to advertisers and investors, Twitter uses a metric that it calls monetizable daily active users, or mDAU, which uses an undisclosed algorithm to count the users that Twitter can make money from and exclude accounts that are automated or otherwise non-monetizable. Twitter also reports the percentage of spam bots on the platform as a portion of mDAU. Twitter says this metric averages around 5%, but you have to take their word for it. According to Musk, that’s a problem.
For months, Musk has complained that Twitter is misleading investors by underreporting the number of bots on the platform. And the billionaire isn’t alone in his complaints: back in August, Twitter’s former security chief and renowned white-hat hacker Pieter ‘Mudge’ Zatko filed a whistleblower complaint accusing Twitter of deliberately underreporting the amount of bots on the platform by only reporting bots as a percentage of mDAU. More whistleblowers were also set to appear at the now-delayed trial between Musk and Twitter.
To prove his point, Musk has leveraged multiple tools to audit the platform for bots. First, Musk tweeted that upwards of 20% of Twitter accounts were bots. Then, Musk’s legal team alleged that 33% of “visible accounts” were bots, based on an audit they conducted with a free online tool whose creator quickly distanced himself from the claim.
20% fake/spam accounts, while 4 times what Twitter claims, could be *much* higher.
My offer was based on Twitter’s SEC filings being accurate.
Yesterday, Twitter’s CEO publicly refused to show proof of <5%.
This deal cannot move forward until he does.
— Elon Musk (@elonmusk) May 17, 2022
Finally, a study commissioned by Musk found that spam and bot accounts make up an estimated 11% of Twitter’s total user base.
For their part, Twitter claims to remove about one million spam bot accounts each day and lock millions more until they have passed anti-spam tests.
Given Musk’s vested interest in making Twitter look bad and Twitter leadership’s opposing interest in presenting metrics that make them look good, who can we trust?
At CHEQ, we’re in the unique position of being able to take a second look; we analyze traffic from over 50,000 websites every day and deploy more than 2,000 cybersecurity challenges on each website visitor to ultimately reveal whether they are a bot/ bad actor or a legitimate user.
So, we checked Elon’s homework.
An 8.55% invalid traffic rate
From a sample base of over 17 million referrals from Twitter collected over the past three quarters from American websites, CHEQ determined that an average of 8.55% of all traffic originating from Twitter was invalid.
Organic traffic showed an invalid rate of 10.43%, nearly twice that of paid which averaged 5.13% invalid, just barely missing Twitter’s own mDAU reporting of 5%.
That’s not surprising – our research consistently finds higher rates of invalid traffic from direct and organic sources, and for their part, Twitter has a strong incentive to crack down on bots committing ad fraud. However, it’s important to note that a user does not need to be monetizable, nor active daily, to cause harm on Twitter. Bots spreading disinformation, for example, have no incentive to click on ads.
Both categories of traffic saw marked increase from Q1 to Q3, with invalid rates for paid traffic climbing 177% from 2.62% to 7.28% while organic invalid traffic increased 145% from 5.81% to 14.26%. That means that the number of bots on Twitter increased since Mr. Musk started tweeting about them.
What can we learn from this data?
We stand by our data, but it’s important to note that no one outside of Twitter’s walled garden can give a full view of the bot issue on the platform. This data only reflects bots who came to websites of CHEQ clients, meaning it is limited in two significant ways: first, it represents a limited sample size, and second, it only represents bots who took a specific action: they clicked a link from Twitter to another web page– most likely an advertising link.
We can’t make any inference that these bots are tweeting false information, or even that they control Twitter accounts, since users and bots can access the website without one. We can safely assume that advertisers aren’t paying Twitter to serve their ads to bots, and a 5.13% invalid rate — trending upwards — is a bad look for a company that derives 90% of its revenue from advertising.
Bots are bigger than Twitter
In truth, the bot problem isn’t really a Twitter problem, it’s an internet problem.
And in handling that problem Twitter, like all online ad platforms, has a conflict of interest. They need to maintain a reputation as a credible ad service that won’t waste ad spend serving ads to bots who can’t convert, but they simultaneously need to present the highest possible active user numbers to attract investors and compete with other social media platforms—and that could disincentivize the company from investigating bots and terminating accounts.
While Twitter is certainly not carrying out a deliberate campaign to defraud advertisers, it’s probably not leveraging all the tools at its disposal to give a transparent view of bots on the platform.
Twitter isn’t the only social media giant with a transparency problem; in 2017, Facebook caught flack for the claim that its platforms could serve ads to 41 million Americans in the coveted 18-24 age bracket, despite the fact that only 31 million Americans in this age group existed.
So whether you think that Elon Musk is a crusader for the greater good of the internet, or a loudmouth trying to back out of a bad deal, he has a point: if businesses are buying ads on the internet, they deserve transparency on who, or what, is actually interacting with those ads.
