local-llm

Local LLM speed measurements from a 4x RTX 3090 rig, with exact launch commands and per-run provenance.

Local RAG app with zero-config Docker setup. FastAPI + Streamlit + Qdrant + Ollama. Just run `docker-compose up --build`! 🚀

AI-Inference-Managed-by-AI: Go binary for managing AI inference on edge devices

Hikari's curated English measured notes on local LLM systems (serving, specdec, training, steering).

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

Fine-tune a local LLM on your own app's data in 15 minutes. Runs entirely on-device, zero cloud after training. Apple Silicon + CUDA.

Introducing Muse Glimmer: open-weight 30B agentic multimodal model that runs on your device (Meta). Interactive local agent lab + guide. Apache 2.0 · on-device AI · function calling

Kairos: An autonomous development supervisor powered by Context Engineering. It provides a living memory and a contextual constitution for your projects to prevent AI context loss.

CLI wrapper for llama.cpp providing an ollama-like experience

Local AI for Apple Silicon. Chat with open LLMs (Qwen, Gemma, DeepSeek, Mistral, Llama…) fully offline on your Mac via Apple MLX — private by design, zero telemetry.

Honeycomb Lab — hex map + OpenAI gateway control plane for a home AI fleet

GPU Service Manager for LLM workloads on Linux/NVIDIA systems.

AI hardware fit calculator for LLM, embedding, reranker, OCR and VLM workloads — VRAM, throughput, licensing and multi-GPU planning

An AI-powered search engine to interact with documentation using RAG and local LLMs. Privately deployable with vector search and a modern frontend.

Run Ornith, Qwen, Nemotron & DeepSeek optimized on AMD Strix Halo — unified OpenAI-compatible server with tuned ROCmFP4 quants, validated 262K context, vision support & hot-swap model zoo (iGPU + NPU)

Gradio-based web UI for medical image interpretation using Google's MedGemma vision-language model.

KV cache middleware for 1M context on 12GB VRAM. Uses K vectors as a retrieval index to fetch V on-demand from system RAM. No model modification, no retraining. Drop-in HuggingFace cache replacement.

豆芽 Douya — 基于 llama.cpp 的隐私优先本地 AI 桌面助手 · 完全离线 · 多后端(CUDA/HIP/SYCL/Vulkan/OpenVINO/CPU) · GPU 加速 · 多模态 · RAG · OpenAI 兼容 API

Run NVIDIA's tri-modal Nemotron Omni (text+vision+audio) entirely on Apple Silicon — pure MLX runtime, parity-verified against NVIDIA's reference

A slim, local-LLM-first AI coding assistant. TUI, Web UI, and desktop app. Runs air-gapped with Ollama, vLLM, or any OpenAI-compatible endpoint.