llm-inference

AI at your fingertips: powerful CLI tools for speech, text, and language processing

大模型学习--从模型部署到模型微调，此项目是经过训练营学习后，结合训练营项目，自我理解消化总结，以及创新型应用。可star/fork

Proxima lets existing GPUs serve 4x more concurrent requests

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models.

Vector compression with TurboQuant codecs for embeddings, retrieval, and KV-cache. 10x compression, pure NumPy core — optional GPU acceleration via PyTorch (CUDA/MPS) or MLX (Metal).

Serve Poolside Laguna S 2.1 (NVFP4) on the NVIDIA DGX Spark (GB10) without hanging your box. Working stack, crash-safe configs, benchmarks, and the exact gotchas.

A Scheduler for Batched LLM Inference

Source code for the paper: Hear Both Sides: Efficient Multi-Agent Debate via Diversity-Aware Message Retention

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

Miscellaneous codes and writings for MLOps

Diagnose vLLM inference servers

Interested in running a conspiracy of agents (technical term) on your Local AI Infra? Who isn't!

Platform for managing LLM fine-tuning (Google Summer of Code '25)

[EMNLP 2025 Findings] MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation

AI-Inference-Managed-by-AI: Go binary for managing AI inference on edge devices

A RAG implementation on Llama Index using Qdrant vector stores as storage. Take some pdfs, store them in the db, use LLM to inference.

Google Colabs to run many Falcon based LLMs

We introduce a large semi-automatically generated dataset of ~400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms that we use to evaluate LLMs

Benchmarks and notes for running modern LLMs with vLLM on 8x Tesla V100-32GB in 2026.

FlashAttention-style custom attention backend for vLLM on AMD MI50/MI60/Radeon VII (gfx906). Downstream fork of mixa3607/ML-gfx906 with replacement HIP kernels and a vllm.general_plugins entry point.