llama-cpp

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable cards.

Natural language control for Python CLI tools using locally-trained SLMs (CPU inference)

A local LLM workhorse that keeps a memory of your workday. On-device meeting notes, dictation, autocomplete, and recall for macOS. GPLv3.

A TUI around llama.cpp for running, managing, and benchmarking local GGUF models and launching the pi coding agent against your local server.

A web-based memory usage and performance calculator for Huggingface GGUF models

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simulation, Godzilla KVarN/TriAttention with SM86/SM89 qualification, CUDA weight sharing, and multi-GPU planning.

Local AI workstation — discover, run, chat, benchmark, and generate images from open-weight models. DFlash/DDTree speculative decoding, TurboQuant & TriAttention cache compression strategies, MLX + llama.cpp + vLLM + MTPLX backends.

Run the LLM Swarm Router on machines to distribute Local Ai to the Swarm - More Machines - MORE SPEED

Run and manage both vLLM and llama.cpp model servers from one terminal UI + CLI — shared Docker Compose profiles, live tok/s, and cross-backend port/GPU conflict checks. NVIDIA + Linux.

An open-source, model-agnostic agent harness for local LLMs. Define agents in YAML (tools, memory, deny-first permissions) and run them against any OpenAI-compatible endpoint: vLLM, Ollama, LM Studio, or llama.cpp.

Locally fine-tuned Russian RAG models: document splitter, query expansion, retrieval embedder. Teacher distillation → GGUF → llama.cpp on AMD Vulkan. Commercial-OK licenses.

A curated list of tools, frameworks, and resources for running AI locally — inference, RAG, orchestration, monitoring, and more.

Rivo Agent is a private, local-first mobile AI assistant that runs compact GGUF language models directly on your phone. It lets users download a compatible model, chat offline, keep local memory, customize assistant behavior, and use AI privately without sending conversations to a remote inference server.

Local LLM speed measurements from a 4x RTX 3090 rig, with exact launch commands and per-run provenance.

AI-Inference-Managed-by-AI: Go binary for managing AI inference on edge devices

Introducing Muse Glimmer: open-weight 30B agentic multimodal model that runs on your device (Meta). Interactive local agent lab + guide. Apache 2.0 · on-device AI · function calling

CLI wrapper for llama.cpp providing an ollama-like experience

Arabic-first expressive zero-shot speech synthesis — Audar-TTS-V1 (Flash + Turbo + Pro). Voice cloning, expression tags, benchmarks, technical report & inference.

AI hardware fit calculator for LLM, embedding, reranker, OCR and VLM workloads — VRAM, throughput, licensing and multi-GPU planning

Local AI model manager