data-processing

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Incremental engine for long horizon agents 🌟 Star if you like it!

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.

High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

Kubernetes-native platform to run massively parallel data/streaming jobs

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

Large-scale pretraining for dialogue

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

Python Stream Processing

Scalable data pre processing and curation toolkit for LLMs

Data and tools for generating and inspecting OLMo pre-training data.

Large-scale pretrained models for goal-directed dialog

Production-ready data processing made easy and shareable

Multi-modal speech separation task data generation script on LRS3 data set.

Framework for processing and filtering datasets

Collects a multimodal dataset of Wikipedia articles and their images

FloatChat AI is a conversational AI platform for ARGO Ocean Data Discovery and Visualization, providing accurate and interactive tools to explore oceanographic data. Built with Python, LangChain, Pandas, and machine learning models, it makes complex ocean data accessible and actionable for researchers and enthusiasts.