vllm

Low-Cost Cross-Domain Web Structured Information Extraction using specialized LoRA adapters.

[ACL 2026] CoCoA: Collaborative Chain-of-Agents for Parametric-Retrieved Knowledge Synergy

htop for your LLM inference cluster

Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System

Proxima lets existing GPUs serve 4x more concurrent requests

A lightweight post-training framework for LLMs and VLMs. 51 algorithms, 38 verified models. Scales with DeepSpeed, vLLM, and Ray.

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models.

RAG using LlamaIndex：Computer Network Q&A System powered by LlamaIndex | 基于 LlamaIndex 框架的计算机网络智能问答系统 - HyDE+混合检索 + vLLM 推理+Ragas评估

Comprehensive, scalable ML inference architecture using Amazon EKS, leveraging Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and MCP

ICE-PIXIU：A Cross-Language Financial Megamodeling Framework

Serve Poolside Laguna S 2.1 (NVFP4) on the NVIDIA DGX Spark (GB10) without hanging your box. Working stack, crash-safe configs, benchmarks, and the exact gotchas.

A curated list of tools, frameworks, and resources for running AI locally — inference, RAG, orchestration, monitoring, and more.

Intelligent load balancer for distributed vLLM server clusters 分布式 vLLM 服务器集群的智能负载均衡器

Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability.

Network-faithful simulation of LLM serving and training deployments

[AAAI 2025 oral] Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit

A Scheduler for Batched LLM Inference

Collection of recent advanced RAG techniques.

An OpenAI Compatible API which integrates LLM, Embedding and Reranker. 一个集成 LLM、Embedding 和 Reranker 的 OpenAI 兼容 API

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.