int4

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

[ICML 2026] GRACE-VLM: deployable INT4 Qwen3-VL via quantization-aware distillation.

Hand-written NVFP4 W4A16 CUDA kernels for Volta