We introduce mGPT, a multilingual variant of GPT-3, pretrained on 61 languages from linguistically diverse 25 language families using Wikipedia and C4 Corpus. We detail the design and pretraining procedure. The models undergo an intrinsic and extrinsic evaluation: language modeling in all languages, downstream evaluation on cross-lingual NLU datasets and benchmarks in 33 languages, and world knowledge probing in 23 languages. The in-context learning abilities are on par with the contemporaneous language models while covering a larger amount of languages, including underrepresented and low-resource languages of the Commonwealth of Independent States and the small peoples in Russia. The source code and the language models are available under the MIT license.

[Paper] [Habr (Russian)] [HugginFace mGPT-1.3B Model Card] [HugginFace mGPT-13B Model Card] [Papers With Code]

pip install -r requirements.txt

The model was pretrained on a 600Gb of texts, mainly from MC4 and Wikipedia.

• MC4
• Wikipedia (version 20201101)

The Wikipedia texts are extracted from the dumps (v. 20201101) with WikiExtractor (Attardi, 2015). Training data was deduplicated, and the text deduplication includes 64-bit hashing of each text in the corpus for keeping texts with a unique hash. We also filter the documents based on their text compression rate using zlib4. The most strongly and weakly compressing deduplicated texts are discarded.

from transformers import GPT2LMHeadModel, GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("sberbank-ai/mGPT")
model = GPT2LMHeadModel.from_pretrained("sberbank-ai/mGPT")

text = "Александр Сергеевич Пушкин родился в "
input_ids = tokenizer.encode(text, return_tensors="pt").cuda(device)
out = model.generate(
        input_ids, 
        min_length=100, 
        max_length=100, 
        eos_token_id=5, 
        pad_token=1,
        top_k=10,
        top_p=0.0,
        no_repeat_ngram_size=5
)
generated_text = list(map(tokenizer.decode, out))[0]
print(generated_text)
Александр Сергеевич Пушкин родился в  г. Санкт-Петербурге.

In general:

eos_token_id=5, 
pad_token=1,
do_sample=True,
top_k=0,
top_p=0.8,
no_repeat_ngram_size=4

English Generation: top_p=0.95, top_k=0

Afrikaans (af), Arabic (ar), Armenian (hy), Azerbaijani (az), Basque (eu), Bashkir (ba), Belarusian (be), Bengali (bn), Bulgarian (bg), Burmese (my), Buryat (bxr), Chuvash (cv), Danish (da), English (en), Estonian (et), Finnish (fi), French (fr), Georgian (ka), German (de), Greek (el), Hebrew (he), Hindi (hi), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Javanese (jv), Kalmyk (xal), Kazakh (kk), Korean (ko), Kyrgyz (ky), Latvian (lv), Lithuanian (lt), Malay (ms), Malayalam (ml), Marathi (mr), Mongolian (mn), Ossetian (os), Persian (fa), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Spanish (es), Swedish (sv), Swahili (sw), Tatar (tt), Telugu (te), Thai (th), Turkish (tr), Turkmen (tk), Tuvan (tyv), Ukrainian (uk), Uzbek (uz), Vietnamese (vi), Yakut (sax), Yoruba (yo)

[mGPT-1.3B Model Card] [mGPT-13B Model Card]

We utilize the DeepSpeed library and Megatron-LM. We pretrain our LMs with a total batch size of 2048 and the context window of 512 tokens. The total number of the training steps is 600k, and the models have seen $400$B tokens during pretraining. The pretraining took 14 days on a cluster of 256 V100 GPUs for mGPT-1.3B and 22 days on 512 V100 GPUs for mGPT-13B.

Habr article about the monoligual mGPT-1.3B models (Russian)

Monolingual models on HuggingFace:

• 🇦🇲 mGPT-1.3B Armenian
• 🇦🇿 mGPT-1.3B Azerbaijan
• 🍯 mGPT-1.3B Bashkir
• 🇧🇾 mGPT-1.3B Belorussian
• 🇧🇬 mGPT-1.3B Bulgarian
• 🌞 mGPT-1.3B Buryat
• 🌳 mGPT-1.3B Chuvash
• 🇬🇪 mGPT-1.3B Georgian
• 🌸 mGPT-1.3B Kalmyk
• 🇰🇿 mGPT-1.3B Kazakh
• 🇰🇬 mGPT-1.3B Kirgiz
• 🐻 mGPT-1.3B Mari
• 🇲🇳 mGPT-1.3B Mongol
• 🐆 mGPT-1.3B Ossetian
• 🇮🇷 mGPT-1.3B Persian
• 🇷🇴 mGPT-1.3B Romanian
• 🇹🇯 mGPT-1.3B Tajik
• ☕ mGPT-1.3B Tatar
• 🇹🇲 mGPT-1.3B Turkmen
• 🐎 mGPT-1.3B Tuvan
• 🇺🇦 mGPT-1.3B Ukranian
• 🇺🇿 mGPT-1.3B Uzbek
• 💎 mGPT-1.3B Yakut

We welcome community contributions to the model, and celebrate both its inference and training technique enhancements.

@article{shliazhko2024mgpt,
 title={mGPT: Few-Shot Learners Go Multilingual},
 author={Shliazhko, Oleh and Fenogenova, Alena and Tikhonova, Maria and Kozlova, Anastasia and Mikhailov, Vladislav and Shavrina, Tatiana},
 journal={Transactions of the Association for Computational Linguistics},
 volume={12},
 pages={58--79},
 year={2024},
 publisher={MIT Press One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA~…}
}