multimodal

An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

LLaVA-Mini is a unified large multimodal model (LMM) that can support the understanding of images, high-resolution images, and videos in an efficient manner.

The Self-Coding System for Your App — Alan AI SDK for React Native

An MCP Multimodal AI Agent with eyes and ears!

Port of MiniGPT4 in C++ (4bit, 5bit, 6bit, 8bit, 16bit CPU inference with GGML)

RAI is a vendor agnostic agentic framework for Physical AI robotics, utilizing ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

Platform for Situated Intelligence

CLIP inference in plain C/C++ with no extra dependencies

Flame is an open-source multimodal AI system designed to translate UI design mockups into high-quality React code. It leverages vision-language modeling, automated data synthesis, and structured training workflows to bridge the gap between design and front-end development.

multimodal routing, geocoding, and map tiles

A hub for various industry-specific schemas to be used with VLMs.

Awesome Multimodal Modeling [Covers MLLM, UMM, and NMM]

Official implementation for "Break-A-Scene: Extracting Multiple Concepts from a Single Image" [SIGGRAPH Asia 2023]

Official code of "EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model"

A simple "Be My Eyes" web app with a llama.cpp/llava backend

Knowledge-Aware machine LEarning (KALE): accessible machine learning from multiple sources for interdisciplinary research, part of the 🔥PyTorch ecosystem. ⭐ Star to support our work!

VisualWebArena is a benchmark for multimodal agents.

[NeurIPS 2025 Spotlight] Official implementation of Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence

TeleMem is a high-performance drop-in replacement for Mem0, featuring semantic deduplication, long-term dialogue memory, and multimodal video reasoning.

Self-hosted Personal AI + agent runtime in .NET (NativeAOT-friendly)