multimodal-deep-learning

[CVPR '23] Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models

[Paperlist] Awesome paper list of multimodal dialog, including methods, datasets and metrics

[MLHC 2025] ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modeling

A set of jupyter notebooks

Transformer-based online speech recognition system with TensorFlow 2

Yet another Ph.D. adventure.

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

MultimodalHugs is an extension of Hugging Face that offers a generalized framework for training, evaluating, and using multimodal AI models with minimal code differences, ensuring seamless compatibility with Hugging Face pipelines.

Multimodal AI Assistant is an advanced chatbot designed for local interactions with LLMs, files, and multimodal functionalities. Built on the Chainlit framework, it integrates Langchain and ChatOllama for hosting models locally, with cloud storage and monitoring via LiteralAI.

A research-oriented training and evaluation framework for ECG-Language Models (ELMs)

Package for bimodal training of deep neural networks on neurological data. Pypi: https://pypi.org/project/BiModNeuroCNN/

ACL 2024 (SRW), Official Codebase of our Paper: "MoExtend: Tuning New Experts for Modality and Task Extension"

PegasusX: The Future of Multimodal Embeddings 🦄 🦄

The repository for the exclusive Daily Paper Club hosted at Agora every 10pm NYC time at this discord: https://discord.gg/Gnzh6dnzyz

Engage in a semantic segmentation challenge for land cover description using multimodal remote sensing earth observation data, delving into real-world scenarios with a dataset comprising 70,000+ aerial imagery patches and 50,000 Sentinel-2 satellite acquisitions.

This repository implements a learning-based beamforming approach leveraging multimodal feature fusion. It includes data preprocessing, augmentation, and a transformer-based network for efficient beam prediction.

Official GitHub repository of the lecture "Multimodal Deep Learning for Recommendation", at the 2024 ACM RecSys Summer School

LeManchot-Analysis is a system for abnormal detection in coupled visible-thermal images

Experiments with multimodal deep learning models based on transformers