evaluation

[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.

This repo is deprecated. Please go to langchain-ai/docs.

Dialectical reasoning architecture for LLMs (Thesis → Antithesis → Synthesis)

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

Automated system for LLM evaluation via agents. Doc as below:

[ICLR'26, NAACL'25 Demo] Toolkit & Benchmark for evaluating the trustworthiness of generative foundation models.

GAN-style self-improvement loop for any text artifact: mutate, grade with a SEPARATE model, keep only verified wins (pairwise-judged), revert the rest. The git history is the improvement log.

Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.

Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation

Data and code for paper "M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models"

⚡ Unit tests for AI. Test prompts, compare models, save money.

A unified evaluation toolkit and leaderboard for rigorously assessing the scientific intelligence of large language and vision–language models across the full research workflow.

This repo contains the code for "MEGA-Bench Scaling Multimodal Evaluation to over 500 Real-World Tasks" [ICLR 2025]

Omni Model Benchmark with high quality and diversity, which reveals the Compositional Law. We’re now focused on Chinese scenarios — and actively seeking partners to co-build English & multilingual versions! Let’s expand global impact together.

Per-collection OCR leaderboards using VLM-as-judge

Evaluation framework for paper "VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?"

PhyX: Does Your Model Have the "Wits" for Physical Reasoning?

3DCF / doc2dataset: token-efficient document layer with NumGuard numeric integrity and multi-framework exports for RAG & fine-tuning.

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

Adaptive Reasoning Engine for Efficient and Context-Aware Intelligence