multimodal-learning

Become a cracked AI/ML researcher/engineer with this unconventional textbook covering maths, computing, and ML with intuition.

Reading list for research topics in multimodal machine learning

An open-source framework for training large multimodal models.

A curated list of Multimodal Related Research.

An official implementation for "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval"

This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"

A Survey on Multimodal Retrieval-Augmented Generation

Knowledge-Aware machine LEarning (KALE): accessible machine learning from multiple sources for interdisciplinary research, part of the 🔥PyTorch ecosystem. ⭐ Star to support our work!

Multi-Modal learning toolkit based on PaddlePaddle and PyTorch, supporting multiple applications such as multi-modal classification, cross-modal retrieval and image caption.

An open source implementation of "Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning", an all-new multi modal AI that uses just a decoder to generate both text and images

Research Trends in LLM-guided Multimodal Learning.

A curated list of awesome Multimodal studies.

Reliable, minimal and scalable library for pretraining foundation and world models

My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"

From scratch implementation of a vision language model in pure PyTorch

[NeurIPS 2024] This repo contains evaluation code for the paper "Are We on the Right Way for Evaluating Large Vision-Language Models"

This repo contains evaluation code for the paper "BLINK: Multimodal Large Language Models Can See but Not Perceive". https://arxiv.org/abs/2404.12390 [ECCV 2024]

Corpus of resources for multimodal machine learning with physiological signals (mmps).

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

Implementation of PALI3 from the paper PALI-3 VISION LANGUAGE MODELS: SMALLER, FASTER, STRONGER"