vllm

A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.

Reliable and Efficient Semantic Prompt Caching with vCache

Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment

Deploy open-source LLMs on AWS in minutes — with OpenAI-compatible APIs and a powerful CLI/SDK toolkit.

This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed.

[ICML 2026] Decoding Tree Sketching (DTS): a training-free & model agonistic & plug-in framework for LLM parallel reasoning.

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

EmbodiedAgents is a fully-loaded ROS2 based framework for creating interactive physical agents that can understand, remember, and act upon contextual information from their environment.

PDF Parsing Tool: GOT's vLLM acceleration implementation, MinerU for layout recognition, and GOT for table formula parsing.

An intelligent tuner for vLLM that automatically monitors GPU metrics, uses Bayesian optimization to tune parameters

Automated Deep Research with LLMs, web search, paper parsing, and didactic summarization.

Open recipes, engine patches, and benchmark harnesses for LLM inference on Intel Arc Pro B60/B70 (Battlemage, Xe2). MoE 35B at 160 t/s decode / 7.5K t/s prefill single-stream, 27B at 50~ t/s decode / 1.7K t/s prefill single stream. vLLM XPU MTP unlocked. Muse Glimmer recipe added!!

A physics-grounded, cost-aware optimization loop for vLLM

A tool for benchmarking LLMs on Modal

LLM-Inference-Bench

Two-node DGX Spark/ASUS GX10 DeepSeek V4 Flash DSpark NVFP4 port for vLLM 0.25, with live dashboard and reproducible deployment.

Sardeenz is a proof-of-concept application that allows you to load more than one model on a given GPU. It allows you to add more and more models onto a GPU, until it is fully utilized.

Turn documents into structured JSON with local-first document AI. Run 100% locally by default, with API, CLI, and Web UI.

LLM-powered security log analyzer: detect threats & anomalies with zero regex — just declare a Pydantic schema. Real-time Telegram alerts, SIEM-ready with Elasticsearch/Kibana. Supports OpenAI, Ollama, vLLM.