tensorrt-llm

A Datacenter Scale Distributed Inference Serving Framework

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

A nearly-live implementation of OpenAI's Whisper.

An Optimized Speech-to-Text Pipeline for the Whisper Model Supporting Multiple Inference Engine

OpenAI compatible API for TensorRT LLM triton backend

Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.

LLM-Inference-Bench

A tool for benchmarking LLMs on Modal

GPU 性能与 AI Infra 学习项目：CUDA/Triton 算子、NCU/NSYS、vLLM/SGLang/TRT-LLM/ms-swift、PyTorch/DeepSpeed/ms-swift 训练、并行架构

大模型推理框架加速，让 LLM 飞起来

Whisper in TensorRT-LLM