data-curation

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Refine high-quality datasets and visual AI models

fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and video datasets. It helps enhance the quality of both images and labels, while significantly reducing data operation costs, all with unmatched scalability.

Scalable data pre processing and curation toolkit for LLMs

🏭 Mega Scale Multimodal DataPipeline for SOTA Foundation Models

Data Curation Tool is a local-first dataset curation, tagging, model-management, downloader, 3D generation, media metadata, and tool-orchestration application. The current documentation set is stored in docs/wiki/

A comprehensive collection of data quality resources, tools, papers, and projects across various data types including traditional data, LLM pretraining/fine-tuning data, multimodal data, and more. Essential reference for researchers and practitioners in data-centric AI.

Turn documents into high-quality instruction datasets with grounding, quality filtering, deduplication, and provenance.