vllm

TopicGPT: A Prompt-Based Framework for Topic Modeling [NAACL'24]

Low latency JSON generation using LLMs ⚡️

Setup and run a local LLM and Chatbot using consumer grade hardware.

Versatile Almost Local, Eventually Reasonable Assistant 🔫

Persist and reuse KV Cache to speedup your LLM.

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

ML software (llama.cpp, ComfyUI, vLLM) builds for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60

Qwen3.5-122B-A10B on DGX Spark: 28.3 → 51 tok/s (+80%)

Implementation for FP8/INT8 Rollout for RL training without performence drop.

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

AI-powered offensive security testing using autonomous agents, directly in your terminal.

Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

LLM 并发性能测试工具，支持自动化压力测试和性能报告生成。

An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.

Home Assistant LLM integration for local OpenAI-compatible services (llamacpp, vllm, etc)

[ICML-2026] Official implementation of "SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience"

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

A CPU Realtime VLM in 500M. Surpassed Moondream2 and SmolVLM. Training from scratch with ease.