multimodal-deep-learning

Implementation of PALI3 from the paper PALI-3 VISION LANGUAGE MODELS: SMALLER, FASTER, STRONGER"

Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!

[ICLR 2026] DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning

Democratization of "PaLI: A Jointly-Scaled Multilingual Language-Image Model"

Your SDK solves all of this. One interface. Unified logic. Local + hosted models. Fine-tuning. Agent tools. Enterprise-ready. Hybrid RAG.Star 🌟 if you like it!

Robust multimodal integration method implemented in PyTorch and TensorFlow

Improving Chest X-Ray Report Generation by Leveraging Warm-Starting

My implementation of Kosmos2.5 from the paper: "KOSMOS-2.5: A Multimodal Literate Model"

Evaluation framework for paper "VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?"

[CVPR 2021] SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events

PyTorch implementation of LIMoE

This repository includes all computer vision, audio, document AI, and multimodal projects.

A Unified Framework for Benchmarking Generative Electrocardiogram-Language Models (ELMs)

[𝐧𝐚𝐭𝐮𝐫𝐞 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐢𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞] ImmunoStruct enables multimodal deep learning for immunogenicity prediction

Implementation of GATO style Generalist Multimodal model capable of image, text, RL and Robotics tasks

Baseline model for multimodal classification based on images and text. Text representation obtained from pretrained BERT base model and image representation obtained from VGG16 pretrained model.

Multimodal generative AI resources : talking heads, STT, TTS, image & video generation, and AI Agent

Easy wrapper for inserting LoRA layers in CLIP.

The open source implementation of the cross attention mechanism from the paper: "JOINTLY TRAINING LARGE AUTOREGRESSIVE MULTIMODAL MODELS"

"Open Source Models with Hugging Face" course empowers you with the skills to leverage open-source models from the Hugging Face Hub for various tasks in NLP, audio, image, and multimodal domains.