llm-inference

✨ AI interface for tinkerers (Ollama, Haystack RAG, Python)

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

Explore the unknown, build the future, own your data.

This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.

LLMs and Machine Learning done easily

Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks

Context-Engine MCP - Agentic Context Compression Suite

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search Engine.

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

PyTorch library for cost-effective, fast and easy serving of MoE models.

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python.

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

A library to communicate with ChatGPT, Claude, Copilot, Gemini, HuggingChat, and Pi

CPU inference for the DeepSeek family of large language models in C++

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.

On-device LLM Inference Powered by X-Bit Quantization

🌱 EcoLogits tracks the energy consumption and environmental footprint of using generative AI models through APIs.

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

Run generative AI models in sophgo BM1684X/BM1688

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.