benchmark

Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.

Practical local LLM recipes and benchmarks for RTX 5060 Ti setups

Accelerated NLP pipelines for fast inference on CPU and GPU. Built with Transformers, Optimum and ONNX Runtime.

A benchmark for AI-driven CAD generation and editing

Deepmark AI enables a unique testing environment for language models (LLM) assessment on task-specific metrics and on your own data so your GenAI-powered solution has predictable and reliable performance.

[EMNLP'21] Visual News: Benchmark and Challenges in News Image Captioning

This repo contains the codes of the penetration test benchmark for Generative Agents presented in the paper "AutoPenBench: Benchmarking Generative Agents for Penetration Testing". It contains also the instructions to install, develop and test new vulnerable containers to include in the benchmark.

[NAACL 2024] MMC: Advancing Multimodal Chart Understanding with LLM Instruction Tuning

Multi-modal AI-generated content detection: image, video, and audio. Benchmarks, training code (DINOv2, DINOv3, ReStraV, BreathNet), and evaluation pipeline for real vs. synthetic classification with calibration-aware metrics.

Handy tool to measure the performance and efficiency of LLMs workloads.

This repo contains the code for "MEGA-Bench Scaling Multimodal Evaluation to over 500 Real-World Tasks" [ICLR 2025]

ICML 2025 - Impossible Videos

Framework for benchmarking fully-managed vector databases

[NeurIPS 2024] Terra: A Multimodal Spatio-Temporal Dataset Spanning the Earth

An benchmark for evaluating the capabilities of large vision-language models (LVLMs)

LLM-Inference-Bench

Open Source AI Benchmarking toolkit for benchmarking speech to text services

Real-event-anchored benchmark for detecting AI-generated videos in real-world crisis settings.

[ICLR 2025] MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs

Vector Index Benchmark for Embeddings (VIBE) is an extensible benchmark for approximate nearest neighbor search methods, or vector indexes, using modern embedding datasets.