apache-spark

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

lakeFS - Data version control for your data lake | Git for data

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

Feathr – A scalable, unified data and AI engineering platform for enterprise

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Spark-Transformers: Library for exporting Apache Spark MLLIB models to use them in any Java application with no other dependencies.