multimodal

Official implementation of JAMIE, Joint variational Autoencoders for Multimodal Imputation and Embedding

Finalist at Brainhack TIL 2024: Team 12000SGDPLUSHIE

Heterogeneous Edge Realtime Measurement and Execution System: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications

A self-hosted omni-modal agentic retrieval platform that understands, searches, and explores documents, images, audio, and video through semantic routing, hybrid retrieval, adaptive agents, and traceable citations.

A research-oriented training and evaluation framework for ECG-Language Models (ELMs)

Disentangled Graph Variational Auto-Encoder for Multimodal Recommendation with Interpretability, IEEE TMM

Codex Skill：让纯文本模型（DeepSeek）借助 StepFun step-3.7-flash 获得看图能力 | Give text-only Codex models (DeepSeek) image understanding via StepFun step-3.7-flash

Collects a multimodal dataset of Wikipedia articles and their images

Learning Adaptive Fusion Bank for Multi-modal Salient Object Detection

The code and weight for LoVA. LoVA is a novel model for Long-form Video-to-Audio generation. Based on the Diffusion Transformer (DiT) architecture, LoVA proves to be more effective at generating long-form audio compared to existing autoregressive models and UNet-based diffusion models.

[ACL 2023] VSTAR is a multimodal dialogue dataset with scene and topic transition information

Miscellaneous codes and writings for MLOps

This open-source project delivers a complete pipeline for converting multi-page documents (PDFs/images) into structured JSON using Vision LLMs on Amazon SageMaker. The solution leverages the SWIFT Framework to fine-tune models specifically for document understanding tasks.

Give non-multimodal Claude Code main models the ability to see pasted screenshots — a ~200-line UserPromptSubmit hook.

CLI & async Python library for free AI chat, image & video generation.

Characterize Anything: A Wondrous Chemical Reaction between vision models and AI Characters

SemEval 2022 Task 5: Multimedia Automatic Misogyny Identification - baseline models and dataset

[EMNLP 2025 Findings] MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation

ACL 2024 (SRW), Official Codebase of our Paper: "MoExtend: Tuning New Experts for Modality and Task Extension"

API/MCP wrapper for RagAnything