inference

An Amazon SageMaker Container for Hugging Face Inference on Graviton and Intel CPUs

在nano-vllm基础上支持moe模型以及Speculative Decoding技术

Extend LLM context windows beyond GPU memory limits with disk-backed KV cache.

Kubernetes-native control plane for scale-to-zero serving of long-tail LLMs

Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

Complete setup guide for a 2-node NVIDIA DGX Spark cluster — distributed training, CUDA inference with EXO, NCCL tuning for Grace Blackwell, NVMe-TCP shared storage, and 200 Gb/s direct fabric networking.

SkyPortal SDK & CLI - Open Source