llm-evaluation

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

Cookbooks and tutorials on Literal AI

An 18 notebook course that isolates and measures each component of agentic loop engineering on real, industry standard software datasets.

Optimize any AI agent’s skills, tools/MCP, and prompts against your own evals.

[MM 2025] A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning

ICLR 2026 | ChemEval: 4-level, 13-dimension, 62-task text/multimodal chemistry benchmark for evaluating LLMs and MLLMs.

A pure-Python structured memory benchmark for multi-agent LLM systems — context graph vs vector RAG vs raw history dump, five scenarios, 18 graded queries, zero API calls.

Synthetic data for fine tuning LLM

Open-source benchmark for evaluating LLMs on 220 real professional tasks across 9 sectors and 44 occupations. Reproducible experiments, artifact validation, grading, and a live evidence dashboard.

面向小团队的一站式 NLP 小模型平台:数据集版本管理 · 训练 · 评估 · 在线部署 · Badcase 闭环,内置大模型 Prompt 盲测评测(人工 + AI 双指标)。

🍯 自进化 AI 能力研究平台：蜜罐系统 → 沙箱评测引擎 → 训练数据集与 LoRA 微调流水线 / Self-Evolving AI Security Research Platform

Runtime detector for reward hacking and misalignment in LLM agents (89.7% F1 on 5,391 trajectories).