Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

Unsupervised text tokenizer focused on computational efficiency

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

Fast and customizable text tokenization library with BPE and SentencePiece support

Ready-made tokenizer library for working with GPT and tiktoken

Explains nlp building blocks in a simple manner.

Byte Pair Encoding for Python!

nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

Fast bare-bones BPE for modern tokenizer training

Go BPE tokenizer (Encoder Decoder) for GPT2 and GPT3

Subword Encoding in Lattice LSTM for Chinese Word Segmentation

Machine Learning for Phishing Website Detection

Simple-to-use scoring function for arbitrarily tokenized texts.

GPT3 encoder & decoder tool written in Swift

Kotlin multiplatform BPE tokenizer library for OpenAI models

High performance unsupervised text tokenization for Ruby

BBPE 底层实现

Learning BPE embeddings by first learning a segmentation model and then training word2vec

Sentiment-based classification for stock article title using PhoBert

✂️ OpenAI's tiktoken tokenizer written in Go