kv-cache-compression

LLM KV cache compression made easy

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.

Native Windows vLLM 0.27.1 wheels: Python 3.13, PyTorch 2.13 + CUDA 13.0, SM 7.5-12.0 for RTX 20/30/40/50, OpenAI-compatible serving, FlashAttention/Rust, 10 KV formats, Multi-TurboQuant, and experimental CPU/RAM/NVMe prompt-KV offload - no WSL or Docker.

A high-throughput LLM serving engine with non-uniform KV cache compression, built on vLLM