inference

Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment

Various installation guides for Large Language Models

A high-performance, universal serving framework for any-to-any models.

whisper-cpp-serve Real-time speech recognition and c+ of OpenAI's Whisper model in C/C++

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

Build an Autonomous Web3 AI Trading Agent (BASE + Uniswap V4 example)

Android native AI inference library, bringing text, image, video, STT, TTS inference

LLM-Inference-Bench

DFlash vLLM for DGX Spark — Plug & Play Block-Diffusion Speculative Decoding

PHP library for interacting with AI platform provider.

run ollama & gguf easily with a single command

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

Arks is a cloud-native inference framework running on Kubernetes

Bleeding edge vLLM Docker image for the NVIDIA DGX Spark (GB10 / sm_121a).

Find the optimal model serving solution for 🤗 Hugging Face models 🚀

First open-source TurboQuant KV cache compression for LLM inference. Drop-in for HuggingFace. pip install turboquant.

ElasticMM: Elastic and Efficient MLLM Serving System

ONNX implementation of the BGE-M3 multilingual embedding model and tokenizer with native C#, Java, and Python implementations. Generates all three embedding types: dense, sparse, and ColBERT vectors.

[MobiCom 2022] InFi is a library for building input filters for resource-efficient inference.

Run, quantize, and fine-tune LLMs on Apple Silicon. Pure Rust, no Python, no CUDA, no ONNX