llm-inference

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

Llama 3+ inference in pure Java

Open-source VMs-as-a-service

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.

LLMFlows - Simple, Explicit and Transparent LLM Apps

GPU environment and cluster management with LLM support

Pure Rust Inference Engine

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo: https://felladrin-minisearch.hf.space

Local AI Assistant

Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.

A command-line interface tool for serving LLM using vLLM.

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.

Self-hosted personalized AI in a mirror.