vllm

Hikari's curated English measured notes on local LLM systems (serving, specdec, training, steering).

TokenPowerBench: Benchmarking the power consumption of LLM inference across engines and deployment configurations

💰 Exocomp Agentic Environment for Go

Automated Triton w8a8 block FP8 kernel tuning tool for vLLM. Auto-detects model architecture, supports Qwen3-Coder-30B-A3B-Instruct-FP8/DeepSeek-V3/custom models, multi-GPU parallel tuning, and generates optimized kernel configs for quantization.

A curated collection of Claude Code agent skills that accelerate the entire vLLM development lifecycle.

Turn any NVIDIA GPU into a local AI platform. Inference + fine-tuning in your browser. One command to start, automatic clustering.

Docker&vLLM官方镜像部署DeepSeek模型，在生产环境中提供类OpenAI接口服务。

A lean, local AI research agent framework that runs on your machine

GLM-5.2 744B with a true 4-bit NVFP4 KV cache on 4x DGX Spark: 42 tok/s peak, 317K-token KV pool (+58.6% vs fp8), needle-verified at 250K depth, serving 316K context

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

Reproducible recipe: serve abliterated Gemma-4-12B (gemma4_unified) at 50-118 tok/s on no-NVLink Blackwell (SM120) via vLLM nightly + ModelOpt FP8/NVFP4 + MTP spec-decode.

An autonomous agentic pipeline that finds, proves, and patches real C memory-safety vulnerabilities end-to-end using a single 7B open-source LLM (Qwen2.5-Coder) on vLLM, with AddressSanitizer as a ground-truth oracle.

FlashAttention-style custom attention backend for vLLM on AMD MI50/MI60/Radeon VII (gfx906). Downstream fork of mixa3607/ML-gfx906 with replacement HIP kernels and a vllm.general_plugins entry point.

Running Large Language Model easily.

LLM, Fine Tuning, Llama 2, Gemma, Mixtral, vLLM, LangChain, RAG, ChromaDB, FAISS

A high-throughput LLM serving engine with non-uniform KV cache compression, built on vLLM

Honeycomb Lab — hex map + OpenAI gateway control plane for a home AI fleet

GPU Service Manager for LLM workloads on Linux/NVIDIA systems.

让 MinerU 在 AMD 显卡上以 vLLM + hybrid-auto-engine 满血运行！ 本项目是对官方文档中 AMD/ROCm 生态支持的有效补充。针对 MinerU 3.x 提供了一套完整的 ROCm 7.x + PyTorch 2.11.0 + vLLM 源码编译与部署方案，解析质量与速度对标 NVIDIA 旗舰显卡。

Run DeepSeek-V4-Flash 304B on a single NVIDIA GB10 / DGX Spark - 2-bit MoE planes, FP4 quality recovery, speculative decoding