evaluation-framework

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

The LLM Evaluation Framework

A framework for few-shot evaluation of language models.

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

Repo for AI Agents The Definitive Guide

Data-Driven Evaluation for LLM-Powered Applications

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

MedEvalKit: A Unified Medical Evaluation Framework

WritingBench: A Comprehensive Benchmark for Generative Writing

The official implementation of ECCV'24 paper "To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now". This work introduces one fast and effective attack method to evaluate the harmful-content generation ability of safety-driven unlearned diffusion models.

A unified evaluation toolkit and leaderboard for rigorously assessing the scientific intelligence of large language and vision–language models across the full research workflow.

Framework to evaluate Trajectory Classification Algorithms

Diagnose the performance of your RAG🩺

Simple customizable evaluation for text retrieval performance of Sentence Transformers embedders on PDFs

BackdoorMBTI is an open source project expanding the unimodal backdoor learning to a multimodal context. We hope that BackdoorMBTI can facilitate the analysis and development of backdoor defense methods within a multimodal context.

Repository for Multililngual Generation, RAG evaluations, and surrogate judge training for Arena RAG leaderboard (NAACL'25)