State-funded social media operators are a hard-to-detect but significant threat to any democracy with free speech, and that threat is growing. In recent years, the extent of these state-funded campaigns has become clear. Russian campaigns undertaken to influence elections are most prominent in the news, but other campaigns have been identified, with the intent to turn South American countries against the US, spread disinformation on the invasion of Ukraine, and foment conflict in America's own culture wars by influencing all sides as part of an effort to weaken America's hegemonic status.

Iranian and Chinese efforts are also well-funded, though not as widespread or aggressive as those of Russia. Even so, Chinese influence is growing, and often it uses social media to spread specific narratives on Xinjiang and the Uyghur situation, Hong Kong, COVID-19, and Taiwan as well as sometimes supporting Russian efforts.

We need better tools to combat this disinformation, both for social media administrators as well as the public. As part of an effort towards that, we have created a proof-of-concept tool that can be operated via browser extension to identify likely state-funded social media operators on Twitter through inference performed on tweet content.

The core of the tool is a DistilBERT language transformer model that has been finetuned on 250K samples of known state operator tweets and natural tweets pulled from the Twitter API. It is highly accurate at distinguishing normal users from state operators (99%), but has some limitations due to sampling recency bias. We intend to iteratively improve the model as time goes on. Currently, a demo of the model is hosted at HuggingFace: state-op-detector

This repo contains all the code used to train the model, as well as links to our cleaned dataset, so that you can replicate the process yourself if desired. We also include Chrome and Firefox extensions that can be used to test the model as you browse Twitter.

• Environment & Setup
  • Dependencies
  • Training Hardware
• Repo Structure
  • Data Folder
  • Notebooks Folder
  • Finetuned Folder
• Usage
  • Retraining
  • Inference
• Data
• Training
• Results
• Limitations
• Contributors

Training was done using pytorch 1.8.2 and transfomers 4.18.0.

Twitter sampling was done with tweepy 4.8.0.

Other dependencies are included in the environment file.

The model was trained on a system with an RTX 3090, and took just under two hours to train with the dataset we've provided.

• The data files we used are too large for Github, so we provide a Google Drive download link here instead. These files should be placed in this folder.

• 1-Tweet-Data-Prep: Data extraction for raw CSVs containing state operator tweets, data cleaning, prep, and sequence concatenation.
• 2-Tweet-Named-Entity-Exploration: Named Entity Recognition for state operator tweets, in order to enhance sampling of normal user tweets (so that the model is exposed to normal user tweets on the same topics frequented by state operators).
• 3-Natural-Tweet-Acquisition: Normal Twitter user tweet sampling, based on entities identified in previous notebook.
• 4-Dataset-Combination-Split: Concatenation and sampling of state operator and natural tweets, and randomized separation into train and test sets.
• 5-Transformer-Finetuning: Finetuning of DistilBERT for tweet classification using the HuggingFace transformers library.

• This folder should contain the trained model. As the files are too large for Github, they can be downloaded here.

If you wish to try training the model, make sure the data is in the appropriate folders and then run the notebook 5-Transformer-Finetuning. Everything needed for training is in this notebook. If you wish to retrain with new or different data, you can choose to modify either the state operator tweet data, the natural tweet data, or both. You can modify this through the appropriate notebook, and then assemble the dataset in 4-Dataset-Combination-Split.

If you wish to try out inference, you can use the model hosted here. This will always be the latest model. You can also download the model from HuggingFace and run it locally:

from transformers import pipeline

classifier = pipeline(model="lingwave-admin/state-op-detector")

classifier("Washington Gives Ankara An Ultimatum | Trump Makes BigTime Overture To Iran | Authoritarian Spirits Congress The Espionage Act And Punishing WikiLeaks | Analysis Of The European Parliamentary Elections | One Mans Quest To Expose A Fake BBC Video About Syria | China Holds Three Trump Cards In War Against US | Maldives Affirms Fealty To Diego Garcia | The End Of Theresa May | Within The Church People Can Become Truly Free | China Hails Modi Victory This Is Why | ")

You can also provide classifier a list of strings, each of which contains a tweet sequence like the above.

However, if you have retrained the model and wish to test inference locally on your own machine (or a hosted platform), you can use the following code from the notebooks folder:

from transformers import AutoModelForSequenceClassification
from transformers import AutoTokenizer
from transformers import TextClassificationPipeline

ft_model = "../finetuned/troll_detect_distilbert"
model = AutoModelForSequenceClassification.from_pretrained(ft_model)
tokenizer = AutoTokenizer.from_pretrained(ft_model)

classifier = TextClassificationPipeline(model=model, tokenizer=tokenizer)

classifier("Washington Gives Ankara An Ultimatum | Trump Makes BigTime Overture To Iran | Authoritarian Spirits Congress The Espionage Act And Punishing WikiLeaks | Analysis Of The European Parliamentary Elections | One Mans Quest To Expose A Fake BBC Video About Syria | China Holds Three Trump Cards In War Against US | Maldives Affirms Fealty To Diego Garcia | The End Of Theresa May | Within The Church People Can Become Truly Free | China Hails Modi Victory This Is Why | ")

The returned value LABEL_1 means that the model thinks the user is a state operator. Otherwise, it will return LABEL_0.

The source data for this project was in the form of raw CSVs provided by Twitter Information Operations. For this project, we used Russian and Chinese datasets from 2019-2021. CSVs should be extracted and placed in a raw_downloads folder in the repo's root directory. We have not included the raw data due to size issues. Natural tweets were obtained through tweepy, by conducting a search for tweets that mention the named entities extracted from the state operator tweets.

All data underwent the following process:

• Removed retweets
• Cleaned text of special characters, @ mentions, newlines, URLs, etc. This is documented in tweet_utils.
• Each tweet was concatenated with the 9 previous tweets from that user to form a 10-tweet sequence in reverse chronological order.
• Tweets were marked as
• Empty tweets, excessively short tweet sequences, etc. were removed.
• The cleaned data was divided into train and test sets and placed in the data/ folder.

The training sequences were further divided into training and validation datasets. We then used the data to finetune the model distilbert-base-uncased from HuggingFace. The training process is documented in notebook 5.

The final model provides us with the following metrics, as evaluated on the test dataset of 31,280 tweet sequences:

• Accuracy: 0.9993
• F1: 0.9985
• Loss: 0.0039

These values are unusually high, but have been found repeatedly with different sampling and data combinations. For a proof of concept this works well, but there are certain questions we would like to investigate in future iterations of the project, including:

• Are there any obvious telltales for natural or state operator tweets that the model is picking up on (e.g., particular word distributions, or the presence or particular odd characters that we didn't notice in our cleaning, or other characteristics of the respective data dumps, like truncation)?
• How big of an effect does recency bias in the natural tweet dataset affect the results? (For example, references to war in Ukraine wouldn't be expected for state operators in 2019-2021, for obvious reasons).

However, there is good reason to think that the model is picking up on valid contrasting characteristics of the two kinds of tweets. A manual examination reveals fairly obvious patterns, including the use of formal title language with occasional English grammar mistakes in state operator tweets, and informal language and slang in natural tweets. The pattern of topics is quite different too--looking back at state operator tweet timelines, we generally find a consistent history of tweets that fulfill a particular agenda.

In general, there are certainly questions to investigate and datasets to refine, but we are confident that even in this iteration the model is picking up on real signals from state operators.

Although this model demonstrates very high accuracy on the validation and test datasets, our data has a recency bias that may have affected some of the results. Specifically, the state operator tweets were mostly from 2019-2021 (with a small proportion scattered in from previous years), whereas the normal user tweets resulted from searches done in May 2022 (and expanded by looking at each users' previous 50 tweets). Thus, the model may be taking recent news topics into account when performing inference.

However, spot checks do seem to show good results even when topics not specific to 2022 or 2019-2021 are mentioned in the tweet strings, so most of the inference likely relies on language style, specific entities, sentiment, etc.

As Twitter releases new datasets of identified state operators, we will test these and see how our model performs. Updated findings will be posted to this readme.

Curt Tigges: data collection & preparation; model training; model deployment

Bryce Meyer: browser extensions

You can reach me at [email protected].