multimodal

From Chain-of-Thought prompting to OpenAI o1 and DeepSeek-R1 🍓

Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model

The first vision plugin for DeepSeek Harness, and the vision bridge for every text-only coding agent. Paste an image, get structured JSON evidence (OCR, layout, semantics). | 全网最强 DeepSeek Harness 外挂视觉插件，为 DeepSeek、GLM 等纯文本模型外挂视觉能力，粘贴图片即得结构化 JSON 证据（OCR、版面、语义）。

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

InternGPT (iGPT) is an open source demo platform where you can easily showcase your AI models. Now it supports DragGAN, ChatGPT, ImageBind, multimodal chat like GPT-4, SAM, interactive image editing, etc. Try it at igpt.opengvlab.com (支持DragGAN、ChatGPT、ImageBind、SAM的在线Demo系统)

An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

Foundation Architecture for (M)LLMs

Represent, send, store and search multimodal data

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

🤖 Type-safe, provider-agnostic TypeScript AI SDK for streaming chat, tool calling, agents, and multimodal apps across OpenAI, Anthropic, Gemini, React, Vue, Svelte, and Solid.

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

Easily compute clip embeddings and build a clip retrieval system with them

Images to inference with no labeling (use foundation models to train supervised models).

streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL

[EMNLP-2024] Build multimodal language agents for fast prototype and production

Comprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation.

Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

mPLUG-Owl: The Powerful Multi-modal Large Language Model Family

HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance