vision

Stream-Omni is a GPT-4o-like language-vision-speech chatbot that simultaneously supports interaction across various modality combinations.

[IEEE TPAMI 2026] Simulating the Real World: Survey & Resources, which contains our survey "Simulating the Real World: A Unified Survey of Multimodal Generative Models" (IEEE TPAMI, 2026) and Awesome-Text2X-Resources. Watch this repository for the latest updates! 🔥

Program that lets you ask questions about your documents, audio, and video files.

Open-source spreadsheets platform for deep research and document processing

This AI Smart Speaker uses speech recognition, TTS (text-to-speech), and STT (speech-to-text) to enable voice and vision-driven conversations, with additional web search capabilities via OpenAI and Langchain agents.

[ICLR'24] Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Multi-Modal Transformer for Video Retrieval

Curiso is an infinite canvas for your thoughts

Pytorch implementation of the models RT-1-X and RT-2-X from the paper: "Open X-Embodiment: Robotic Learning Datasets and RT-X Models"

Official code for Paper "Mantis: Multi-Image Instruction Tuning" [TMLR 2024 Best Paper]

Recrafting Video Ads with Generative AI

为 Claude Code 赋能多模态视觉能力，支持豆包、通义千问、GPT-4o 等模型，用于截图 / UI / 图表分析；适配 DeepSeek 等无视觉底座，搭配 browser-harness 可做前端布局自动化检查。

Easiest way of fine-tuning HuggingFace video classification models

Agentic RAG, Multi-Agent Systems, and Vision Reasoning are three pipelines to find the perfect LLM

PyTorch tutorials A to Z

Convert PowerPoint files into semantically rich text using vision language models

The PyVisionAI Official Repo

An open source chat bot architecture for voice/vision (and multimodal) assistants, local(CPU/GPU bound) and remote(I/O bound) to run.

Awesome Audio-Visual Intelligence, Survey of Audio-Visual Intelligence

Near-native image understanding for DeepSeek Harness