inference-optimization

High-efficiency floating-point neural network inference operators for mobile, server, and Web

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

A physics-grounded, cost-aware optimization loop for vLLM

A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models.

A high-throughput LLM serving engine with non-uniform KV cache compression, built on vLLM