local-llm

A TUI around llama.cpp for running, managing, and benchmarking local GGUF models and launching the pi coding agent against your local server.

Distributed peer-to-peer LLM inference. Your prompt never leaves your device in clear text.

Uncensor Qwen 3.8 27B with a pre-baked system pack for Ollama, plus GGUF download links.

Production-grade recipe for DeepSeek-V4-Flash-0731 (284B MoE) on 2x NVIDIA DGX Spark: self-healing 2-node vLLM cluster, reboot-verified, tuned DSpark speculative decoding (~75 tok/s), full benchmarks, OpenAI Codex CLI integration

Truthful scheduler-step and streaming visualization for local LLM inference

Local LLM chat app for Apple Silicon Macs.

VL-JEPA inspired pipeline — compress images/text locally via Ollama, send compact payloads to any LLM API. Cut token costs by ~80%.

One Mac process. Many models. Real speed. Multi-model LLM serving with prefix reuse, MTP acceleration, and OpenAI APIs — built for Apple Silicon, measured against mlx-lm and llama.cpp.

Run the LLM Swarm Router on machines to distribute Local Ai to the Swarm - More Machines - MORE SPEED

Run and manage both vLLM and llama.cpp model servers from one terminal UI + CLI — shared Docker Compose profiles, live tok/s, and cross-backend port/GPU conflict checks. NVIDIA + Linux.

Open-source, self-hosted AI workspace for local and open-weight LLMs with vLLM, LiteLLM, autonomous agents, MCP tools, deep research, artifacts, and BYOK.

Local CLI for Google's TranslateGemma translation models with multi-platform support (MLX for Apple Silicon, PyTorch for CUDA/CPU).

An open-source, model-agnostic agent harness for local LLMs. Define agents in YAML (tools, memory, deny-first permissions) and run them against any OpenAI-compatible endpoint: vLLM, Ollama, LM Studio, or llama.cpp.

Run Qwen3.5 on your iPhone. No cloud. No API keys. No subscriptions. 100% on-device.

Writing that's kin to yours

Local AI load balancer for Ollama and MLX fleets. Auto-discovery, smart routing, OpenAI-compatible API, zero config. Perfect for Mac Minis & Studios.

Local inference server for Apple Silicon — hot-swaps MLX models (LLM, vision, embeddings, TTS, STT) via OpenAI API

I'm sorry, Dave. I'm afraid I can't let that spam call through. — Local AI answering service with voice cloning, call recording, and sub-second latency. Runs entirely on your GPU. No cloud. No per-minute billing. Just HAL.

Local-first AI coding agent runtime — your models, fail-closed safety, evidence-gated self-improvement

vLLM + Qwen3.6-27B (BF16) OpenAI-compatible inference server on AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). Vision input, 256K context, /v1/responses with separated reasoning, via TheRock ROCm.