triton-inference-server

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

OpenAI compatible API for TensorRT LLM triton backend

Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.

Deploy stable diffusion model with onnx/tenorrt + tritonserver

Build Recommender System with PyTorch + Redis + Elasticsearch + Feast + Triton + Flask. Vector Recall, DeepFM Ranking and Web Application.

Miscellaneous codes and writings for MLOps

Deploy KoGPT with Triton Inference Server