distributed-inference

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

Achieve state of the art inference performance with modern accelerators on Kubernetes

SGLang-Omni empowers high-performance serving for TTS, ASR, speech and omni models.

Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.

An imperative command-line-interface for AI workload orchestration