multimodal

Experiments and data for the paper "When and why vision-language models behave like bags-of-words, and what to do about it?" Oral @ ICLR 2023

Official Repository of paper VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding

Code and Pretrained Models for ICLR 2023 Paper "Contrastive Audio-Visual Masked Autoencoder".

LLMInternSkill: LLM internship resume and job-search Codex Skill for resume polish, JD tailoring, evidence guard, interview grilling, and Project Scout. 大模型实习简历与求职工具箱。

Prompts of GPT-4V & DALL-E3 to full utilize the multi-modal ability. GPT4V Prompts, DALL-E3 Prompts.

Official repo for "GeoVista: Web-Augmented Agentic Visual Reasoning for Geolocalization"

Phi-3, -3.5, and -4 for Mac: Locally-run Vision and Language Models for Apple Silicon

Archived snapshot of Thinking-with-Visual-Primitives

My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"

Code for the CVPR 2024 paper highlight and demo "PIGEON: Predicting Image Geolocations".

DeepEar ｜ 顺风耳 An open-source framework for Deep Research and Financial Signal Tracking. 一个用于深度研究与金融信号追踪的开源框架。

🔍 OpenSearch-VL provides a fully open recipe for training strong multimodal deep search agents through high-quality data curation, diverse visual/search tools, and fatal-aware agentic reinforcement learning.

Code/Data for the paper: "LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding"

Multi-Modal Transformer for Video Retrieval

[EMNLP2024 Demo], [ICASSP 2025], [ICASSP 2026] A user-friendly library for reproducible video moment retrieval and highlight detection. It also supports audio moment retrieval.

llama.cpp (GGUF LLMs) and llava.cpp (GGUF VLMs) for ROS 2

Awesome-Biomolecule-Language-Cross-Modeling: a curated list of resources for paper "Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey"

(AAAI 2024) BLIVA: A Simple Multimodal LLM for Better Handling of Text-rich Visual Questions

From scratch implementation of a vision language model in pure PyTorch

Language Models Can See: Plugging Visual Controls in Text Generation