kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Sardeenz is a proof-of-concept application that allows you to load more than one model on a given GPU. It allows you to add more and more models onto a GPU, until it is fully utilized.