multimodal

MultiModal Classifier Hierarchy (MMoCHi)

This repository implements a learning-based beamforming approach leveraging multimodal feature fusion. It includes data preprocessing, augmentation, and a transformer-based network for efficient beam prediction.

Build your own local-first digital lifeform: an AI companion with memory, soul, multimodal perception, proactive dialogue, and auditable execution.

A comprehensive framework to explore whether embodied multimodal models are plausibly resilient

EgoNormia | Benchmarking Physical Social Norm Understanding in VLMs

Multimodal recorder for activity recognition

A PyTorch implementation of the paper "MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis".

A Multimodal Discord bot with machine learning functions, including LLM chat, Image generation, and Speech Generation capabilities

Context Runtime — a database query planner for LLM context. Decides what a model sees before it answers; plans it, runs it through reused substrate, and learns from the outcome.

Open source implementation of a vision transformer that can understand Videos using max vit as a foundation.

This repository contains practical AI projects applying machine learning techniques to real-world problems.

[COLING 2024] TIGER: A Unified Generative Model Framework for Multimodal Dialogue Response Generation

A curated list of papers, benchmarks, project pages, and code for proactive agents.

Self-hosted multimodal AI workspace — chat, vision QA, text-to-image, image-to-image in one conversation

Multimodal and multilingual topic model with pretrained embeddings

[NeurIPS2023] LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering

Unified-modal Salient Object Detection via Adaptive Prompt Learning

It is a chatbot based on LangChain.

[Reproduce] Code for the EMNLP2018 paper "A Visual Attention Grounding Neural Model for Multimodal Machine Translation".

Exploring Visual Interpretability for Contrastive Language-Image Pretraining