llm-ops

🦍 The API and AI Gateway

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

Enterprise-grade, commercial-friendly agentic workflow platform for building next-generation SuperAgents.

Julep — durable, composable AI agents. Flows that crash and resume, retry safely, and explain every step.

AutoRAG: Now your agent can find anything in your computer. It gets smarter if you are using it frequently.

RAG (Retrieval Augmented Generation) Framework for building modular, open source applications for production by TrueFoundry

The platform for LLM evaluations and AI agent testing

The collaborative spreadsheet for AI. Chain cells into powerful pipelines, experiment with prompts and models, and evaluate LLM responses in real-time. Work together seamlessly to build and iterate on AI applications.

AIConfig is a config-based framework to build generative AI applications.

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

China's 1st enterprise multi-agent IT ops platform. LLM-powered auto-remediation for Zabbix/Prometheus. Docker deploy.

OpenTelemetry-native LLM observability CLI. Point any OTEL exporter at it and watch your LLM/agent traces in real time.

High-performance Ollama & LM Studio proxy with per-user fair-share queuing, round-robin scheduling, and a real-time TUI dashboard. Built in Rust.

[DEPRECATED] Moved to microsoft/agent-governance-toolkit

A living map of the AI agent security ecosystem.

A production-grade control layer that sits between your application logic and any LLM — input validation, schema enforcement, circuit breaking, targeted retry, and audit logging in one composable pipeline.

Static analysis for AI agent configs, tool descriptions, and system prompts — catches vague tool descriptions, missing stop conditions, and schema gaps before they reach runtime. Zero-LLM, deterministic checks, built for CI.

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

Multi-model AI agent runtime. Define agents in YAML, route each role to a model, orchestrate with 7 patterns (ReAct, Plan & Execute, Fan-Out, Pipeline, Supervisor, Swarm, Glyph), and deploy as a REST/WebSocket API with RAG, memory, MCP tools, guardrails and OpenTelemetry observability.

Miscellaneous codes and writings for MLOps