{rtiktoken} is a thin wrapper around
tiktoken-rs (and in turn
around OpenAI’s Python library
tiktoken). It provides functions
to encode text into tokens used by OpenAI’s models and decode tokens
back into text using
BPE tokenizers. It
is also useful to count the numbers of tokens in a text to guess how
expensive a call to OpenAI’s API would be. Note that all the
tokenization happens offline and no internet connection is required.
Another use-case is to compute similarity scores between texts using tokens.
Other use-cases can be found in the OpenAI’s cookbook How to Count
Tokens with
tiktoken.
To verify the outputs of the functions, see also OpenAI’s Tokenizer Platform.
You can install rtiktoken like so:
# Dev version
# install.packages("devtools")
# devtools::install_github("DavZim/rtiktoken")
# CRAN version
install.packages("rtiktoken")library(rtiktoken)
# 1. Encode text into tokens
text <- c(
"Hello World, this is a text that we are going to use in rtiktoken!",
"Note that the functions are vectorized! Yay!"
)
tokens <- get_tokens(text, "gpt-4o")
tokens
#> [[1]]
#> [1] 13225 5922 11 495 382 261 2201 484 581 553 2966 316
#> [13] 1199 306 38742 8251 2488 0
#>
#> [[2]]
#> [1] 12038 484 290 9964 553 9727 2110 0 115915 0
# 2. Decode tokens back into text
decoded_text <- decode_tokens(tokens, "gpt-4o")
decoded_text
#> [1] "Hello World, this is a text that we are going to use in rtiktoken!"
#> [2] "Note that the functions are vectorized! Yay!"
# Note that it's not guaranteed to produce the identical text as text-parts
# might be dropped if no token match is found.
identical(text, decoded_text)
#> [1] TRUE
# 3. Count the number of tokens in a text
n_tokens <- get_token_count(text, "gpt-4o")
n_tokens
#> [1] 18 10The different OpenAI models use different tokenizers (see also source
code of
tikoken-rs
for a full list).
The following models use the following tokenizers (note that all functions of this package both allow to use the model names as well as the tokenizer names):
| Model Name | Tokenizer Name |
|---|---|
| GPT-4o models | o200k_base |
ChatGPT models, e.g., text-embedding-ada-002, gpt-3.5-turbo, gpt-4- |
cl100k_base |
Code models, e.g., text-davinci-002, text-davinci-003 |
p50k_base |
Edit models, e.g., text-davinci-edit-001, code-davinci-edit-001 |
p50k_edit |
GPT-3 models, e.g., davinci |
r50k_base or gpt2 |
rtiktoken is built using extendr and Rust. To build the package,
you need to have Rust installed on your machine.
rextendr::document()
devtools::document()