tiktoken

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

Online playground for OpenAPI tokenizers

go version of tiktoken

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

A fast RWKV Tokenizer written in Rust

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust