evaluation

Production-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python + TypeScript.

🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

Evaluate results from ASR/Speech-to-Text quickly

This repo contains evaluation code for the paper "MileBench: Benchmarking MLLMs in Long Context"

Latxa: An Open Language Model and Evaluation Suite for Basque

Building Systems with the ChatGPT API

Synthetic data for fine tuning LLM

A starter kit for evaluating benchmarks on the 🤗 Hub

[ICLR 2026🔥] SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense

ESGenius: EMNLP 2025 Main Oral benchmark for LLM ESG and sustainability knowledge, with 1,136 questions, evaluation code, interactive heatmap, and Hugging Face dataset release.

VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation [TMLR26]

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

An MCP server for dataset generation, fine-tuning, RL, and evaluation of LLMs — directly from your coding agent.

A curated full-stack LLM engineering playbook — from AI infrastructure and model lifecycle to agents, MCP, memory, and evaluation.

GenAI - Building AI Workflows | Agentic Appls | Working with LLMs | HuggingFace | OpenAI | Gemini

WMB-100K — The first 100,000-turn benchmark for AI memory systems

项目描述：为解决电商客服在物流、售后等高频咨询场景下，人工重复查询、知识定位慢及信息需反复核对等痛点，搭建 Agent 智能客服系统。通过意图分流、业务工具调用与 RAG 知识库检索，完成订单物流查询、售后信息收集和商品问答，提升响应效率与回答准确性。

A comprehensive framework to explore whether embodied multimodal models are plausibly resilient

Project page for AV-Phys Bench: Do Joint Audio-Video Generation Models Understand Physics?