llm-inference

All the code and materials

Data Center and Client workload and software optimizations for Intel hardware.

Personal private Chatbot powered by Amazon Bedrock LLMs with a data analytics feature that provides isolated servereless compute on for data analysis. Chatbot features web-based intuitive user interface that can be accessed from any device (laptops, phones, etc.), handles multi-modal document upload and chat and maintains privacy of conversations.

One-command vLLM installation for NVIDIA DGX Spark with Blackwell GB10 GPUs (sm_121 architecture)

Gradio based tool to run opensource LLM models directly from Huggingface

Nano vLLM with vLLM v1's request scheduling strategy and chunked prefill

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more.

Granite Switch — Build AI models like you build software

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

Enterprise-grade LLM automated deployment tool that makes AI servers truly "plug-and-play".

Hand-written NVFP4 W4A16 CUDA kernels for Volta

Completion After Prompt Probability. Make your LLM make a choice

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.

This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed.

Flowchart-like UI to interconnect LLM's and Huggingface models, and deploy them as a REST API with little to no code.

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

VindexLLM is a pure Delphi, GPU-powered LLM inference engine that uses Vulkan compute shaders to run GGUF models entirely on the GPU. It performs full transformer inference without relying on Python, CUDA, or other external runtimes, requiring only vulkan-1.dll, which is typically included with modern GPU drivers.

A curated, hands-on library of notebooks, demos, and resources for AI/ML, Deep Learning, Generative AI, AI-Agents, fine-tuning, and modern tooling.

[SOICT 2024] LLM-Powered Video Search: A Comprehensive Multimedia Retrieval System

Android native AI inference library, bringing text, image, video, STT, TTS inference