multi-modality

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

Parsing-free RAG supported by VLMs

Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!

[CVPR 2023] Collaborative Diffusion

An open-source implementation for training LLaVA-NeXT.

[CVPR 2025 Highlight] Official code for "Olympus: A Universal Task Router for Computer Vision Tasks"

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

[CVPR'24] RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech

Embed arbitrary modalities (images, audio, documents, etc) into large language models.

An open-source cloud-native of large multi-modal models (LMMs) serving framework.

An all-new Language Model That Processes Ultra-Long Sequences of 100,000+ Ultra-Fast

Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!

My implementation of Kosmos2.5 from the paper: "KOSMOS-2.5: A Multimodal Literate Model"

[ICLR 2025] Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations, ICCV 2021