vllm

演示 vllm 对中文大语言模型的神奇效果

AI Coding Assistant CLI for offline enterprise environments - Local LLM platform with Plan & Execute architecture, Supervised Mode, and auto-update system

Serve Llama 3.3 70B (with AWQ quantization) using vLLM and deploy it on BentoCloud.

A REST API for vLLM, production ready

Policy-driven seamless lazy loading

Open-source local implementation of MiniMax H3 Context-IR. Turn simple prompts into structured, validated H3 video briefs.

[CVPR2025] Official Repository for IMMUNE: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment

An open-source, AI-driven Social Media Digital Twin powered by LLMs (Ollama, vLLM) for Computational Social Science simulations, network analysis, and human-agent interaction.

Experimental RAG playground for exploring retrieval quality, corpus construction, and filter-chain design. Features configurable ranking and filtering pipelines, visual document grounding, chat interfaces, web search, Open WebUI integration, and rich debugging insights. Ollama and vLLM, running in Dev Containers or natively on Linux and Windows.

A web-based memory usage and performance calculator for Huggingface GGUF models

A hybrid router that uses Spot GPU instances to reduce costs and Serverless GPUs for making Cold Starts faster.

面向开发者与初学者的 nano-vLLM 交互式源码教程 - 通过 13 个 HTML 互动实验 + 13 章中文教程理解大模型推理引擎

a simple lightweight large language model pipeline framework.

Dockerized LLM inference server with constrained output (JSON mode), built on top of vLLM and outlines. Faster, cheaper and without rate limits. Compare the quality and latency to your current LLM API provider.

Orpheus TTS Server with streaming support (TTFB ~160ms)

This project is the backend engine for a fully autonomous AI-powered call center. It integrates a large language model (LLM), speech recognition, and text-to-speech to manage real-time phone conversations via Asterisk.

A "standard library" of Triton kernels.

Official implementation for Text Generation Beyond Discrete Token Sampling

Patched native-Windows build of vLLM. Three Windows-specific fixes (CPU-relay for Gloo, Qwen3 reasoning parser, wildcard model name) on top of SystemPanic 0.19.0. Engine for devnen/qwen3.6-windows-server.

Production-grade recipe for DeepSeek-V4-Flash-0731 (284B MoE) on 2x NVIDIA DGX Spark: self-healing 2-node vLLM cluster, reboot-verified, tuned DSpark speculative decoding (~75 tok/s), full benchmarks, OpenAI Codex CLI integration