multimodal-learning

[ICLR 2026] DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning

[NAACL 2022 Findings] Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction

A professional list on Multi-Modalities For Time Series Analysis (MM4TSA) Papers and Resource.

Quick Long Video Understanding [TMLR2025]

[CVPR 2023] Official code repository for "How you feelin'? Learning Emotions and Mental States in Movie Scenes". https://arxiv.org/abs/2304.05634

PhyX: Does Your Model Have the "Wits" for Physical Reasoning?

Efficient Multimodal Foundation Model Adaptation for Recommendation

A Unified Framework for Benchmarking Generative Electrocardiogram-Language Models (ELMs)

[𝐧𝐚𝐭𝐮𝐫𝐞 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐢𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞] ImmunoStruct enables multimodal deep learning for immunogenicity prediction

[Paperlist] Awesome paper list of multimodal dialog, including methods, datasets and metrics

"Open Source Models with Hugging Face" course empowers you with the skills to leverage open-source models from the Hugging Face Hub for various tasks in NLP, audio, image, and multimodal domains.

[MLHC 2025] ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modeling

Code for "Dual-Level Adaptive Incongruity-Enhanced Model for Multimodal Sarcasm Detection".

[NeurIPS 2025] TL;DR: Aligning pretrained unimodal models with the proposed framework using limited paired data yields ~52% gains in cross-modality zero-shot classification and ~92% in retrieval.

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

[ICIP 2025] Official implementation of RT-X Net: RGB-Thermal cross attention network for Low-Light Image Enhancement

Collects a multimodal dataset of Wikipedia articles and their images

Torch code for Visual Question Generation

Unsupervised specificity-guided optimization of Image Captioning models to encourage meaningful diversity in the generated captions. Code for the paper Generating Diverse and Meaningful Captions: Unsupervised Specificity Optimization for Image Captioning (Lindh et al., 2018).

My implementation of the model KosmosG from "KOSMOS-G: Generating Images in Context with Multimodal Large Language Models"