data-quality

Learn how to develop, deploy and iterate on production-grade ML applications.

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

Always know what to expect from your data.

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Refine high-quality datasets and visual AI models

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

The Open Source Feature Store for AI/ML

lakeFS - Data version control for your data lake | Git for data

Learn how to design, develop, deploy and iterate on production-grade ML applications.

An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈

The Virtual Feature Store. Turn your existing data infrastructure into a feature store.

Feathr – A scalable, unified data and AI engineering platform for enterprise

Scalable data pre processing and curation toolkit for LLMs

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

🏭 Mega Scale Multimodal DataPipeline for SOTA Foundation Models

An open-source pipeline for training natural language understanding models

A comprehensive collection of data quality resources, tools, papers, and projects across various data types including traditional data, LLM pretraining/fine-tuning data, multimodal data, and more. Essential reference for researchers and practitioners in data-centric AI.