vision-and-language

Research code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training"

PyTorch code for "VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks" (CVPR2022)

This repo contains evaluation code for the paper "BLINK: Multimodal Large Language Models Can See but Not Perceive". https://arxiv.org/abs/2404.12390 [ECCV 2024]

Official implementation of X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

VLM driven tool that processes surveillance videos, extracts frames, and generates insightful annotations using a fine-tuned Florence-2 Vision-Language Model. Includes a Gradio-based interface for querying and analyzing video footage.

Recent advancements propelled by large language models (LLMs), encompassing an array of domains including Vision, Audio, Agent, Robotics, Fundamental Sciences such as Mathematics, and Ominous.

PyTorch code for “TVLT: Textless Vision-Language Transformer” (NeurIPS 2022 Oral)

[EMNLP'21] Visual News: Benchmark and Challenges in News Image Captioning

[MICCAI 2024, top 11%] Official Pytorch implementation of Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography

🥶Vilio: State-of-the-art VL models in PyTorch & PaddlePaddle

Pytorch code for ICRA'21 paper: "Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation"

GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024)

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

[ACL 2021] Learning Relation Alignment for Calibrated Cross-modal Retrieval

[ICLR 2025] Official code repository for "TULIP: Token-length Upgraded CLIP"

Code for Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense? [COLM 2024]

[ACL 2023] Delving into the Openness of CLIP

We introduce new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance.

Grounding Language Models for Compositional and Spatial Reasoning