tokenizers

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

NeuroRVQ: Multi-Scale Biosignal Tokenization for Generative Foundation Models

Develop DL models using Pytorch and Hugging Face

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

[Unofficial] Simple .NET wrapper of HuggingFace Tokenizers library

Multi-arch bindings for @huggingface/tokenizers.