tokenizer

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

A small library for converting tokenized PHP source code into XML (and potentially other formats)

Language model tokenization at GB/s

Online playground for OpenAPI tokenizers

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING

Ready-made tokenizer library for working with GPT and tiktoken

Text2Text Language Modeling Toolkit

Full-stack open-source AI engine for building language models — tokenizer training, transformer architecture, cognitive reasoning and chat pipeline.

CLI tool – estimates LLM tokens/costs and runs provider-aware load tests for OpenAI, Anthropic, OpenRouter, or custom endpoints.

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

Open Source Generative AI Models for Automatic Rhythm Game Beatmap Generation (for acoustic people)

On Device Stable Diffusion In Mobile Devices

A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

A fast RWKV Tokenizer written in Rust

NeuroRVQ: Multi-Scale Biosignal Tokenization for Generative Foundation Models

[ICML'25] MedTok: Multimodal Medical Code Tokenizer

ONNX implementation of the BGE-M3 multilingual embedding model and tokenizer with native C#, Java, and Python implementations. Generates all three embedding types: dense, sparse, and ColBERT vectors.

A tiny language model that talks like a house cat named Miso.