Stanford Researcher Open-Sources Gigatoken, Boosting Tokenization Speed

Tokenization is a critical early step in training and running large language models, converting raw text into units that models can process. As datasets expand, that stage can become a costly preprocessing bottleneck even before intensive model computation begins. Faster tokenizers with broad model compatibility could shorten data preparation cycles and reduce the computing resources required to encode large text corpora.

Stanford University doctoral student Marcel Rød has open-sourced Gigatoken, a BPE tokenizer written in Rust. Benchmark figures released with the project show throughput reaching 24.53 GB per second, making it as much as 989 times faster than HuggingFace Tokenizers. Gigatoken currently supports 23 widely used large language models, including GPT-2, Llama and DeepSeek, positioning the library as a high-speed option for large-scale text encoding and preprocessing.