evaluation

A unified evaluation framework for large language models

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

YAO = Yielding AI Outcomes. A rigorous engineering, evaluation, governance, and portability system for reusable agent skills.

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.

Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!

Avalanche: an End-to-End Library for Continual Learning based on PyTorch.

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.

(IROS 2020, ECCVW 2020) Official Python Implementation for "3D Multi-Object Tracking: A Baseline and New Evaluation Metrics"

WFGY is heading toward WFGY 5.0 Polaris Protocol, a major open-source release for AI reasoning, RAG, agents, and real-world workflows. Includes Problem Map, Global Debug Card, WFGY 4.0, and the CFV Easter Egg.

心理健康大模型 (LLM x Mental Health), Pre & Post-training & Dataset & Evaluation & Depoly & RAG, with InternLM / Qwen / Baichuan / DeepSeek / Mixtral / LLama / GLM series models

A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.

Building blocks for rapid development of GenAI applications

The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

Evaluate your LLM's response with Prometheus and GPT4 💯

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.