inference

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

MLX Studio - Home of JANG_Q - Image Gen/Edit + Chat/Code All in one - + OpenClaw (Anthropic API)

SGLang-Omni empowers high-performance serving for TTS, ASR, speech and omni models.

Example notebooks for working with SageMaker Studio Lab. Sign up for an account at the link below!

AI Productivity Tool - Free and open source, improve user productivity, and protect privacy and data security. Including but not limited to: built-in local exclusive ChatGPT, DeepSeek, Phi, Qwen and other models, one-click batch intelligent processing of pictures, videos, audio, etc.

One API for all LLMs either Private or Public (Anthropic, Llama V2, GPT 3.5/4, Vertex, GPT4ALL, HuggingFace ...) 🌈🐂 Replace OpenAI GPT with any LLMs in your app with one line.

Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.

Accurate, large-scale, and extensible simulator for LLM inference Systems

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Segmind Distilled diffusion

Real-time inference for Stable Diffusion - 0.88s latency. Covers AITemplate, nvFuser, TensorRT, FlashAttention. Join our Discord communty: https://discord.com/invite/TgHXuSJEk6

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

⚠️ Legacy repository for Geti v2.x. For Geti v3.0+, visit https://github.com/open-edge-platform/geti

sparkrun - launch, manage, and stop LLM inference workloads on NVIDIA DGX Spark systems

LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.

A project that optimizes OWL-ViT for real-time inference with NVIDIA TensorRT.

Guideline following Large Language Model for Information Extraction

https://wavespeed.ai/ Context parallel attention that accelerates DiT model inference with dynamic caching

Super performant RAG pipelines for AI apps. Summarization, Retrieve/Rerank and Code Interpreters in one simple API.