data-centric-ai

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Refine high-quality datasets and visual AI models

The data scientist's open-source choice to scale, assess and maintain natural language data. Treat training data like a software artifact.

🏭 Mega Scale Multimodal DataPipeline for SOTA Foundation Models

Intriguing Properties of Data Attribution on Diffusion Models (ICLR 2024)

A comprehensive collection of data quality resources, tools, papers, and projects across various data types including traditional data, LLM pretraining/fine-tuning data, multimodal data, and more. Essential reference for researchers and practitioners in data-centric AI.

Implementation of TSDS: Data Selection for Task-Specific Model Finetuning. An optimal-transport framework for selecting domain-specific and task-specific training data to improve LLM finetuning and instruction tuning.