kvcache

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Persist and reuse KV Cache to speedup your LLM.

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

Sardeenz is a proof-of-concept application that allows you to load more than one model on a given GPU. It allows you to add more and more models onto a GPU, until it is fully utilized.

inference cookbook / inference 框架原理解析