-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathDESCRIPTION
21 lines (21 loc) · 1.04 KB
/
DESCRIPTION
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Package: rtiktoken
Title: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
Version: 0.0.6
Authors@R: c(
person("David", "Zimmermann-Kollenda", , "[email protected]", role = c("aut", "cre")),
person("Roger Zurawicki", role = "aut", comment = "tiktoken-rs Rust library"),
person("Authors of the dependent Rust crates", role = "aut", comment = "see AUTHORS file")
)
Description: A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
License: MIT + file LICENSE
URL: https://davzim.github.io/rtiktoken/, https://github.com/DavZim/rtiktoken/
BugReports: https://github.com/DavZim/rtiktoken/issues
Suggests:
testthat (>= 3.0.0)
SystemRequirements: Cargo (Rust's package manager), rustc >= 1.65.0
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.2
Config/rextendr/version: 0.3.1
Config/testthat/edition: 3
Config/rtiktoken/MSRV: 1.65.0