multimodal

This repo contains the original implementation of VAuLT, the Vision-and-Augmented-Language Transformer. We provide instructions to download some multimodal social-media datasets, and scripts to experiment with. VAuLT is a stack of Transformers, a LM like BERT that preprocesses the text input of ViLT

[IEEE TMM 2024] IGReg: Image-Geometry-Assisted Point Cloud Registration via Selective Correlation Fusion

Open Translator: Speech To Speech and Speech to text Translator with voice cloning and other cool features

This is the official repository for CardioLab. A machine and deep learning framework for the estimation and monitoring of laboratory abnormalities throught ECG data.

An unofficial implementation of the CVPR 2020 paper Multimodal Categorization of Crisis Events in Social Media

Zero-dependency Python SDK for WeChat Bot (iLink protocol) — text, image, video messaging & MCP server | 零依赖微信机器人 Python SDK，支持多媒体消息和 MCP 服务器

AI eyes that roll through video footage — watch, understand, act

Road Maps as Free Geometric Priors: Weather-Invariant Drone Geo-Localization with GeoFuse

🖼️ Workshop: Build a multimodal AI agent with Haystack & GPT-4o — featuring image understanding, document retrieval, conversational memory, and human-in-the-loop safety controls

Universal MCP server for logistics. Connect any TMS/WMS to AI agents. Shipments, carriers, tenders, tracking — all via Model Context Protocol.

Frontend interface for building chat based system and connecting with agent driven workflows.

Semantic alignment of astronomical data with natural language using multi-modal models. (Jax) Code associated with https://arxiv.org/abs/2403.08851 (COLM 2023).

The open source implementation of the multi grouped query attention by the paper "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"

https://vaila.readthedocs.io/

[WACV 2025] I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting

WWW'24, Mirror Gradient (MG) makes multimodal recommendation models approach flat local minima easier compared to models with normal training.

A comprehensive survey of Vision–Language Models: Pretrained models, fine-tuning, prompt engineering, adapters, and benchmark datasets

A low-token visual evidence compiler for text-only coding agents. Convert images into compact Visual Evidence Packets (VEP) for DeepSeek, Codex, Claude Code, and other text-only models.

It is a voice bot based on LLM.

AI-powered multimodal chat app with real-time responses, file support, token tracking, and dark mode. Built with Next.js. Open source under MIT.