vllm-serve

Repo for vLLM Hook, an vLLM plug-in for programming internal states of models deployed on vLLM

Proxima lets existing GPUs serve 4x more concurrent requests

Intelligent load balancer for distributed vLLM server clusters 分布式 vLLM 服务器集群的智能负载均衡器

Diagnose vLLM inference servers

Distributed Inference with vLLM