multimodal

Code for Deep Learning for Modern AI

Multimodal Instruction Tuning for Llama 3

[IEEE TMI 2024] MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images

An implementation of Deep Generalized Canonical Correlation Analysis (DGCCA or Deep GCCA) with pytorch.

Toy-scale unified multimodal model experiments — encoder-free understanding & generation with Mixture-of-Transformers on MLX/Apple Silicon

A Unified Framework for Benchmarking Generative Electrocardiogram-Language Models (ELMs)

AI-PP3 is a command-line tool that uses artificial intelligence to analyze RAW photos and generate optimized processing profiles (PP3 files) for RawTherapee.

Combining ViT and GPT-2 for image captioning. Trained on MS-COCO. The model was implemented mostly from scratch.

Developer tools to debug and build realtime voice agents. Supports multiple models.

[𝐧𝐚𝐭𝐮𝐫𝐞 𝐦𝐚𝐜𝐡𝐢𝐧𝐞 𝐢𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞] ImmunoStruct enables multimodal deep learning for immunogenicity prediction

State-of-the-art embedding models fine-tuned for the ecommerce domain. +67% increase in evaluation metrics vs ViT-B-16-SigLIP.

🐝 | From Data to Prognosis: Embedding Multimodal Oncology Data for Precision Medicine

A modular, high-performance Retrieval-Augmented Generation framework with multi-path retrieval, graph extraction, and fusion ranking

Multimodal Hashtag Prediction with instagram data & pytorch (2nd Place on OpenResource Hackathon 2019)

中文领域的多模态Bert

Multimodal Chat with Gemini API

DiffBlender: Scalable and Composable Multimodal Text-to-Image Diffusion Models

ALICE and its prior work, Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality

'Talk to your slide deck' (Multimodal RAG) using foundation models (FMs) hosted on Amazon Bedrock and Amazon SageMaker

[ICML'25] MedTok: Multimodal Medical Code Tokenizer