Fake Twitter Trends Fool Turks For Over Five Years. Here’s How
Almost Half of the top 5 Twitter trends in Turkey and 20% of the top 10 world trends are fake — created by fake or compromised accounts. Turkish Twitter users wake up every morning to a Twitter trend that promotes a phishing website to compromise legitimate users.
Some of the accounts that were used to push fake trends were disclosed by Twitter’s Election Integrity Team on 12 June 2020. Nonetheless, the problem persists.
Here, I explain the method the attackers use, how they cleverly use compromised but otherwise benign Twitter accounts for their aims and the success of the fake trends.
The paper is accepted to the IEEE European Symposium on Security and Privacy 2021 and is available here: https://arxiv.org/abs/1910.07783
The Method
Fake news and social media manipulation wreak havoc in Turkey which has a highly polarized society. I’m a researcher in disinformation — it’s literally my job to understand it — and even I sometimes struggle to distinguishing what’s truth and what’s fake or who’s a person that I can keep a meaningful conversation with and who’s a troll. Unfortunately, Twitter’s efforts so far have fallen short in this brave new world. Along with other problems (I’ll get to them slowly in this blog), attackers easily manipulate and push any keyword they want to Twitter trends, conveying their message to an audience of 12.7 millions. Such “trends” include not only political campaigns but also hate speech, profanity, marriage proposals (yes, you’ve read that right), asking for votes to a Survivor (TV show) contestant, gambling websites, illegal football streaming websites and many more!
Unfortunately, the only news articles documenting this problem in Turkish media are from 2014. Even more surprising is that the problem — an ongoing and a prevalent attack — is never mentioned in the academic literature.
Top trends are visible on Twitter’s main page and the Explore section of Twitter app and you can click on them to see which tweets containing them. If you click on a fake trend, most of the times you’ll either see no tweets at all, or a few spam tweets that embed (hijack) the trends to grab attention. You may also find some people complaining about it (“Why this is in the trends?”). Such tweets appear to be posted after the keyword reaches trending. In reality, if a keyword is trending, there has to have been a spike of tweets (i.e. many tweets containing the keyword in a short time, i.e. within a minute) that contain it the keyword for the trending algorithm to consider it as a trend. So how come there are hardly any tweets for a top trend and all appear to be posted after the keyword reach trending?
Here’s an example from August 18th, 2020, today! twitteragt açıldı is ranking 4th in the trends with only three tweets! You can observe all three patterns and there are no other tweets!
It turns out that there was a spike of activity promoting the trend, but the tweets within the spike were deleted so quickly that you cannot see them when you search for them. However, Twitter provides 1% of all tweets in real time so anyone can see 1% of deleted tweets and the exact time they were deleted. Internet archive’s Twitter Stream Grab stores this sample dataset since 2011, and luckily, I found an external database of trends to study this problem in detail.
Here’s an example of a trend which advertised a cafe in Samsun, Turkey and the number of tweets that are created (•’s) and deleted (×’s) per minute:
Notice that except for the spikes of creation and deletion of tweets, there are hardly any tweets. However, the keyword stayed in the top trends for hours.
The number of tweets within the spike seems low (30 tweets) but this is because we have only 1% of all tweets. It’s impossible to get the exact number of tweets without full access to Twitter stream. The highest number of attack tweets I observed was 2900 tweets within one minute.
Examining the tweets used for the trend manipulation, I found out that their content is random and does not mean anything, i.e. “trigonometry apple (fruit) #ExampleTrend to cycle”. The verbs are in infinitive form, which is very rare in Turkish language, and the parenthesis are used for disambiguation. I believe they are randomly picked from some sort of a list of Turkish words (maybe from a dictionary). Early (before 2015) versions of this attack involved tweets with random string of letters and the trend (i.e. “#ExampleTrend sdfsf”) or with premade statements (i.e. “#ExampleTrend oh yeah this trend is nice”) or with snippets from literary works (i.e. “#ExampleTrend to be or not to be”). I suspect Twitter got better in detecting those or those tweets were costly to create so attackers came up with this “lexicon method” to trick Twitter’s spam filters.
Moving on to the accounts that are employed for trend manipulation, I found something even more shocking. Not all users tweeting such gibberish were fake accounts or bots, some were actually humans just like you and me. They share cat videos, they comment on the news, they conversate with their friends. Some had put their instagram accounts on Twitter profile’s homepage, so I contacted them via DM there and found that they were either not aware of the situation or were aware but helpless and did not think it was a big problem. The hackers were controlling their accounts only to fake trends and for nothing else. Otherwise, owners continune to use their accounts. The hackers were like a parasite. This strategy make sense as Twitter is probably more conservative in suspending such accounts.
So what makes them tweet in their behalves? It is not a Twitter app, as attack tweets’ metadata shows that they originate from official Twitter apps, i.e. Twitter for iPhone. Thus, compromised users might have leaked their passwords or installed malicious 3rd party apps on their device. Attackers push a trend which promotes a phishing website to Twitter trends every morning as I mentioned before. Check trends in Turkey at like 9 a.m. (GMT +3) and you’ll see some 2gram that contain “Twitter” in it. They change it everyday. Today it was twitterragt açıldı. Here’s the attacker promoting it:
Suspicious of this website, I signed up a fake account and immediately gained 15 followers (yaay!). However, my account quickly became part of the botnet and participate in 580 attacks in 6 months, until I decided that it’s enough and logged out from the suspicious session (a Linux machine from Istanbul). I believe some of the users are compromised in this way but not all of them, though I’ve yet to identify other sources of compromisation.
There were also other interesting patterns such as accounts that were dormant for years and/or have absolutely no tweets (but actively tweeting and deleting to manipulate trends), accounts that have a full profile (name, surname, homepage, profile photo of a human) but otherwise only had retweets so it is hard to argue they are humans or not; and, weirdly enough, a bunch of K-Pop fans, which have a mixture of tweets and retweets. Botometer failed to classify these accounts as bots except for dormant accounts with some tweets (it gives error when the account has no tweets) perhaps because it is biased towards such accounts. I do not think any bot detection method that takes a single user as input and does not work in realtime can catch these bots due to the deletions.
Lastly, I noticed one compromised user complaining that someone is tweeting from his account from different cities. It turns out that the attackers controlling also spoof the attack tweets’ locations (geotags). You can see the same user tweeting from Istanbul and an hour late from Antalya, beaching. They might be doing this to ensure that the keyword trends nationwide. Alternatively, Twitter’s trending algorithm may be favoring the trends that are tweeted from various locations.
So the method of the attackers in a nutshell is:
- Tweet using many accounts (one account per tweet) to give the impression that a keyword is important and should trend, then delete the tweets to quickly hide the fact that the trend is fake. Obviously Twitter’s trending algorithm does not take into account the fact that the tweets have been deleted — or this attack would fail.
- Inject tweets that would pass the spam filters of Twitter but not necessarily the Turing test — humans will not see these gibberish tweets anyway since they are quickly deleted.
- Employ compromised but otherwise benign accounts to build trust with Twitter’s trending algorithm.
- Employ fake locations to make the keyword trend nationwide.
The Research
As a Turkish citizen who got used to trend manipulation on Twitter and also someone with data science background, I initially thought these findings were nothing special and I could publish this as a hobby project, i.e. explain my observations and come up with an alternative bot detection method to solve the problem then let it go. However, my colleagues who come from other backgrounds and are knowledgeable about the political climates of other countries recognized this as a serious security problem. Together we teamed up to analyze the attacks in detail using historical data. Classifying randomly generated tweets is very easy as they have a fixed number of tokens, do not start with an uppercase letter, do not have punctuation, and all are deleted. Therefore, we could classify trends as fake or not with 99% accuracy and F-score. Since we only have 1% of all tweets, we might have missed some trends that are fake, because their data do not exist in this dataset. So all the numbers I report here are lower bounds.
Here are the findings in a nutshell:
The attacks started in June 2015 and are still ongoing by today! There are over 19,000 unique keywords pushed by these attacks. The daily average of the top 5 trends in Turkey is 47.5% and 26.7% for the top 10. For the world, it is 13.7% for the top 5 trends and 19.7% for the top 10. Here’s a nice plot:
Note that the time period we chose as a sample to study is mostly after the last election (2019 Istanbul Election rerun) in Turkey. Imagine the situation before the elections!
Almost all fake trends enter the trends list in the top 10 and majority of them enter in the top 5. This is not surprising as only the top 10 trends are visible from the main page of Twitter and attackers are aiming for visibility.
The fake trends stay in the trends list much longer, even when compared to other trends that also enter the trends list in top 10.
It takes roughly 5 minutes for a fake trend to be trending. (Twitter trends are refreshed every 5 minutes we can say they are trending instantly.)
In 70% of cases, attackers tweeted and deleted all their tweets before their target keyword reach trends list and there is absolutely no other tweet containing that keyword besides attackers’. This means that the attackers create the “trends” from scratch!
The attacks can successfully push a target keyword to trends 95 out of 100 times. We are not always certain why they fail in these 5% of cases. At times the attackers do make a mistake, i.e. typing the target keyword twice like #tugrulcanelmas#tugrulcanelmas or targetting “Tree Fest” but pushing “Tree” to trending instead. It appears that there are also cases where Twitter filtered the keyword i.e. #PornLoverJohnDoe.
We identified 108,000 bots employed in these attacks from June 2015 to September 2019. 45% of these bots are now suspended or deleted.
We could not find such attacks in anywhere else in the world (Turkish hackers deserve a cookie!). The BBC reported about trend manipulation in Saudi Arabia in 2018, so we checked the bots promoting the fake trend that the BBC reported was purchased and found that Saudi attackers were using the premade statement method (trend + religious quote or some slogan like long live the king) that was abandoned by Turks long ago in 2014 (the Saudis have long way to go.) The Saudis ask for $200 to get a topic to trend, which we find too expensive, since the Turks will do it for only 170TL ($20!). Don’t believe us? Check those Twitter trends! Yes, of course, they advertise their “Trending topic service” using manipulated Twitter trends!
Here, I summarized the severity of the problem as best as I could. I’ll stop here and leave the kind of trends they manipulate to my next post.
If you wanna read more details about the methodology or are interested in even more results, refer to our publication. If you have something to discuss, please do not hesitate to leave a comment or reach me on Twitter!
This study is conducted by Tuğrulcan Elmas, Dr. Rebekah Overdorf, Ahmed Furkan Özkalay and supervised by Prof. Karl Aberer who runs LSIR in EPFL, Switzerland. It is supported in part by the Open Technology Fund’s Information Controls Fellowship.
