sglang

gpt_server是一个用于生产级部署LLMs、Embedding、Reranker、ASR、TTS、文生图、图片编辑和文生视频的开源框架。

Automated system for LLM evaluation via agents. Doc as below:

Efficient LLM inference on Slurm clusters.

Local diagnostic CLI for NVIDIA DGX Spark (GB10). Detects power caps, UMA pressure, thermal risk, CUDA 13/SM_121 wheel mismatches, Docker issues, and validates vLLM/Ollama/llama.cpp/SGLang recipes.

Agent and LLM API performance monitoring via network packet probe. Measures performance of OpenClaw, Claude, Codex, DeepAgents and more — deployed on the provider side, no SDK changes required.

A tool for benchmarking LLMs on Modal

Arks is a cloud-native inference framework running on Kubernetes

A high-performance RDMA distributed file system for fast LLM Inference and GPU Training.

Benchmarking Open-Ended Inference Optimization by AI Agents

GPU 性能与 AI Infra 学习项目：CUDA/Triton 算子、NCU/NSYS、vLLM/SGLang/TRT-LLM/ms-swift、PyTorch/DeepSpeed/ms-swift 训练、并行架构

An imperative command-line-interface for AI workload orchestration

DeepSeek-V3, R1 671B on 8xH100 Throughput Benchmarks

htop for your LLM inference cluster

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models.

Network-faithful simulation of LLM serving and training deployments

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

AI-Inference-Managed-by-AI: Go binary for managing AI inference on edge devices

Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

inference cookbook / inference 框架原理解析