inference

htop for your LLM inference cluster

⚡ A seamless integration of HuggingFace Transformers & Diffusers with RBLN SDK for efficient inference on RBLN NPUs.

Comprehensive, scalable ML inference architecture using Amazon EKS, leveraging Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and MCP

A curated list of tools, frameworks, and resources for running AI locally — inference, RAG, orchestration, monitoring, and more.

Intelligent load balancer for distributed vLLM server clusters 分布式 vLLM 服务器集群的智能负载均衡器

Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability.

Network-faithful simulation of LLM serving and training deployments

Deferred Continuous Batching in Resource-Efficient Large Language Model Serving (EuroMLSys 2024)

Official GitHub repository for Phison aiDAPTIV. NAND cache memory-boosted AI inference and LLM fine-tuning platform.

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.

H.E.I.M.D.A.L.L looks at fleet telemetry and gives you natural-language insights. GPU data loading (cuDF), local LLM inference (Gemma 2), and production NIM on GKE. Open the notebooks, run cells, get answers! Quick start should not take longer than 10 minutes and the T4 path is completely free!

🚀 The Ultimate Curated List of LLMOps Tools, Frameworks, and Resources - A comprehensive collection of the best tools for Large Language Model Operations

Dual-mode cloud burst LLM router. Edge-first or cloud-first inference with data sovereignty, cost controls, and automatic failover. https://aiburstcloud.com

Hands-on GPU/HPC infrastructure operations: K8s GPU scheduling, HAMi sharing, Slurm, observability & vLLM inference. Learn it free on a laptop; validate on one cheap GPU.

TokenPowerBench: Benchmarking the power consumption of LLM inference across engines and deployment configurations

Turn any NVIDIA GPU into a local AI platform. Inference + fine-tuning in your browser. One command to start, automatic clustering.

Know your VRAM before you run. Instant GPU memory estimates for any HuggingFace model.

LLM inference engine built from scratch in C++. No PyTorch, no frameworks.

Benchmark OpenAI-compatible AI endpoints and AI Accelerators in a reproducible structured way

Collection of ChatGPT alternatives & LLM tuning methods