multimodal

[MICCAI 2023] Fundus-Enhanced Disease-Aware Distillation Model for Retinal Disease Classification from OCT Images

A Go toolkit for building AI agents and applications across multiple providers. Unified LLM client, agent framework with handoffs, tool calling, streaming, structured output, MCP integration, and cost tracking.

Multimodal foundation model predicting transcriptome-wide virtual spatial transcriptomics from histology.

ACM MM'23: A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation

Inverse DALL-E for Optical Character Recognition

Train YOLO + VLM with one command. Auto-generate vision-language training data from YOLO labels - no extra labeling needed.

A Multimodal Transformer: Fusing Clinical Notes With Structured EHR Data for Interpretable In-Hospital Mortality Prediction

[NeurIPS 2024] Official Repository of Multi-Object Hallucination in Vision-Language Models

The open source implementation of the cross attention mechanism from the paper: "JOINTLY TRAINING LARGE AUTOREGRESSIVE MULTIMODAL MODELS"

A instruction data generation system for multimodal language models.

Self-hosted AI agent workspace with tool calling, MCP, multi-model routing, sandboxed execution, and multi-agent workflows. Web and desktop.

Build and explore multimodal web interactives with pieces of paper!

[Paperlist] Awesome paper list of multimodal dialog, including methods, datasets and metrics

[CVPR '23] Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models

Tool to automatically generate text descriptions for images using Ollama vision models (LLaVA, Qwen3-VL, Llama Vision)

Edge-native AI API gateway — cost-optimized routing across providers, multi-protocol support, built on Cloudflare Workers.

"Open Source Models with Hugging Face" course empowers you with the skills to leverage open-source models from the Hugging Face Hub for various tasks in NLP, audio, image, and multimodal domains.

Build use cases with VideoDB

A simple POC of FastRTC, a framework to use voice mode in python!

Audio-Visual Corruption Modeling of our paper "Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring" in CVPR23