tokenization

💫 Industrial-strength Natural Language Processing (NLP) in Python

🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

Language model tokenization at GB/s

Easy token price estimates for 400+ LLMs. TokenOps.

A suite of image and video neural tokenizers

Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.

Ready-to-run examples showing NoPII PII protection with OpenAI, Anthropic, LangChain, LlamaIndex, and more

C++ Library for tokenizing MIDI files, designed to be compatible with the MIDITok python library

Convert Hugging Face tokenizers to ONNX models for cross-language compatibility (.NET, Java, Python) with embedding models

A character tokenizer for Hugging Face Transformers

Advance information retrieval system that combines advanced indexing, machine learning, and personalized search to enhance academic research and document discovery.