vllm

The Continuous Verification and Optimization layer for self-hosted LLMs

SAM — Smart Agentic Model: CLI coding agent for open-source LLMs. pip install sam-agent

AI hardware fit calculator for LLM, embedding, reranker, OCR and VLM workloads — VRAM, throughput, licensing and multi-GPU planning

CI scripts designed to build a Pascal-compatible version of vLLM.

Batch LLM Inference with Ray Data LLM: From Simple to Advanced

Open-source local AI server configs, GFX906 runtime maintenance, reproducible benchmarks, and QC methods for affordable AI research infrastructure.

An out-of-tree vLLM plugin for Mobilint NPU runtime integration.

LLM inference engine built from scratch in C++. No PyTorch, no frameworks.

vLLM tool parser for Qwen2.5-Coder models using <tools> tag format.

FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inference

GLM-5.2 on 4x DGX Spark with adaptive MTP K2/K4/K5, FULL CUDA graphs, DCP2, 520K context, and a downloadable ARM64 runtime image.

Distributed Inference with vLLM

Benchmark OpenAI-compatible AI endpoints and AI Accelerators in a reproducible structured way

DeepSeek-V4-Flash-0731 tuned on 4x DGX Spark (GB10) at TP=4 — 123.13 tok/s decode. Every variable measured one at a time, with the values that lost.

nvtop for vLLM — an interactive terminal dashboard for vLLM serving performance (concurrency, throughput, cache & KV memory, latency, spec-decode, GPU)

A slim, local-LLM-first AI coding assistant. TUI, Web UI, and desktop app. Runs air-gapped with Ollama, vLLM, or any OpenAI-compatible endpoint.

Terraform setup for deploying a private coding LLM on Vast.ai with vLLM, Qwen3 Coder, and OpenCode.

在nano-vllm基础上支持moe模型以及Speculative Decoding技术

GLM-5.2 QuantTrio (Unpruned ) on 4xDGX Spark (GB10): 327K-655K context or up to 5 concurrent agents, one receipe. TP4 + DCP + MTP

Extend LLM context windows beyond GPU memory limits with disk-backed KV cache.