llm-inference

An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

FineTune LLMs in few lines of code (Text2Text, Text2Speech, Speech2Text)

One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.

fully local, temporally aware natural language file search on your pc! even without a GPU. find relevant files using natural language in less than 1 second.

[arxiv: 2503.23895] Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancement

The Next-Gen Database for AI—an infrastructure designed for data and AI. As the MySQL of the AI era.

A tool that transforms audio or video files into text transcripts and generates concise meeting minutes. Stay organized and efficient in your meetings, and get ready for Phase 2 where we'll be open for contributions to enable real-time meeting transcription!

Run any Large Language Model behind a unified API

AI stack for interacting with LLMs, Stable Diffusion, Whisper, xTTS and many other AI models

🦙 Free and Open Source Large Language Model (LLM) chatbot web UI and API. Self-hosted, offline capable and easy to setup.

A blazingly fast, privacy first & OPEN AI Chat Interface

Self-hosted framework for orchestrating fleets of specialist AI agents — ensemble reasoning and a full agentic coding pipeline, model-agnostic and local-friendly.

GLM-5.2-NVFP4-REAP-469B serving on SM120 (4× RTX PRO 6000 Blackwell) — one-command vLLM launch recipe, 250K context, DeepSeek Sparse Attention + MTP speculative decode

⚡️ A fast and flexible PyTorch inference server that runs locally, on any cloud or AI HW.

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

Large Language Model (LLM) Inference API and Chatbot

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching