llm-inference

Open recipes, engine patches, and benchmark harnesses for LLM inference on Intel Arc Pro B60/B70 (Battlemage, Xe2). MoE 35B at 160 t/s decode / 7.5K t/s prefill single-stream, 27B at 50~ t/s decode / 1.7K t/s prefill single stream. vLLM XPU MTP unlocked. Muse Glimmer recipe added!!

Distributed 56M-parameter LLM inference across 3 ESP32-S3 boards via ESP-NOW , Split-PLE + KV cache, fully offline.

🚀 A powerful Flutter-based AI chat application that lets you run LLMs directly on your mobile device or connect to local model servers. Features offline model execution, Ollama/LLMStudio integration, and a beautiful modern UI. Privacy-focused, cross-platform, and fully open source.

An endpoint server for efficiently serving quantized open-source LLMs for code.

vLLM Router

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages.

run ollama & gguf easily with a single command

CompanionLLM - A framework to finetune LLMs to be your own sentient conversational companion

This is the official implementation for the paper: Use Your INSTINCT: INSTruction optimization usIng Neural bandits Coupled with Transformers

[ICPP'25] TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference

vLLM Qwen 3.6-27B (AWQ-INT4) + DFlash speculative decoding on AMD Strix Halo (gfx1151 iGPU, 128 GB UMA, ROCm 7.13). 24.8 t/s single-stream, vision, tool calling, 256K context, OpenAI-compatible, Docker. Matches DGX Spark FP8+DFlash+MTP at a third of the cost. No CUDA.

Ollama with intel (i)GPU acceleration in docker and benchmark

Testing speed and accuracy of RAG with, and without Cross Encoder Reranker.

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

# Ollama<=>Ollama Inference Cluster

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any OpenAI-compatible local server, emitting JSONL traces, inspect reports, and stable exit codes.

This is an automated content pipeline that scrapes, curates, and summarizes data from a source, then generates and publishes ready-to-post content to Telegram channels, X (Twitter), and Instagram accounts. It also creates and uploads videos to YouTube.

Qurio brings multi-provider models, custom agents, reusable skills, MCP servers, HTTP tools, retrieval, long-term memory, Deep Research, and Scrapbook into one operating surface.

Collection of Basic Prompt Templates for Various Chat LLMs (Chat LLM 的基础提示模板集合)

A high-performance Llama3 implementation using Micronaut and GraalVM Native Image