data-engineering

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

lakeFS - Data version control for your data lake | Git for data

DeepAnalyze is the first agentic LLM for autonomous data science. 🎈你的AI数据分析师，自动分析大量数据，一键生成专业分析报告！

Data Science Roadmap from A to Z

SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.

Maestro: Netflix’s Workflow Orchestrator

Blazing-fast Data-Wrangling toolkit

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

Learn how to design, develop, deploy and iterate on production-grade ML applications.

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Learn to build your Second Brain AI assistant with LLMs, agents, RAG, fine-tuning, LLMOps and AI systems techniques.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine

A new SOTA for RAG — an original retrieval architecture and an open-source knowledge base for humans and agents.

Python Stream Processing

Feathr – A scalable, unified data and AI engineering platform for enterprise

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

ktx is an executable context layer for data and analytics agents 🐙 Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company