model-serving

A high-throughput and memory-efficient inference and serving engine for LLMs

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

A framework for efficient model inference with omni-modality models

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

Open-Source Personal Cloud OS for Always-On Agents

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

Community maintained hardware plugin for vLLM on Ascend

OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

AICI: Prompts as (Wasm) Programs

The simplest way to serve AI/ML models in production

SGLang-Omni empowers high-performance serving for TTS, ASR, speech and omni models.

Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

The missing bridge between your ML models and your AI agents.