multimodal

Official implementation of the paper "FLIP: Cross-domain Face Anti-spoofing with Language Guidance". (ICCV 2023)

Democratization of "PaLI: A Jointly-Scaled Multilingual Language-Image Model"

第五届百度西安交大大数据竞赛 城市区域功能分类 Baseline

Awesome-3D/Multimodal-Anomaly-Detection-and-Localization/Segmentation/3D-KD/3D-knowledge-distillation

Neural Machine Translation with universal Visual Representation (ICLR 2020)

AI‑powered web exam system with face recognition, webcam, browser, and audio checks ensures integrity in remote assessments. Flask prototype reached ~72% functional accuracy and 90%+ detection rates, reducing cheating in online education.

Create high-fidelity Codex companions from 2+ photos with local open-weight neural editing and no OpenAI API key.

A professional list on Multi-Modalities For Time Series Analysis (MM4TSA) Papers and Resource.

[ICLR 2024] Official repository for "Vision-by-Language for Training-Free Compositional Image Retrieval"

An open source chat bot architecture for voice/vision (and multimodal) assistants, local(CPU/GPU bound) and remote(I/O bound) to run.

Multi-modal speech separation task data generation script on LRS3 data set.

Robust multimodal integration method implemented in PyTorch and TensorFlow

Multi-modal AI-generated content detection: image, video, and audio. Benchmarks, training code (DINOv2, DINOv3, ReStraV, BreathNet), and evaluation pipeline for real vs. synthetic classification with calibration-aware metrics.

Multimodal AI agent, an interactive data studio with on-demand ML inference, media generation, and a database explore

Unofficial implementation and experiments related to Set-of-Mark (SoM) 👁️

Multimodal entity linking (MEL) aims to utilize multimodal information to map mentions to corresponding entities defined in knowledge bases. We release three MEL datasets: Weibo-MEL, Wikidata-MEL and Richpedia-MEL, containing 25,602, 18,880 and 17,806 samples from social media, encyclopedia and multimodal knowledge graphs respectively. A MEL dataset construction approach is proposed, including five stages: multimodal information extraction, mention extraction, entity extraction, triple construction and dataset construction. Experiment results demonstrate the usability of the datasets and the distinguishability between baseline models.

Deep Research through Multi-Agents, using GraphRAG

[NeurIPS 2024] GenRL: Multimodal-foundation world models enable grounding language and video prompts into embodied domains, by turning them into sequences of latent world model states. Latent state sequences can be decoded using the decoder of the model, allowing visualization of the expected behavior, before training the agent to execute it.

Open-source multimodal dataset: 98K+ clinical cases & 139K+ medical images from PubMed Central

The official implementation of 'Align and Attend: Multimodal Summarization with Dual Contrastive Losses' (CVPR 2023)