attention-mechanisms

Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT

Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling

PyTorch Implementation of Jamba: "Jamba: A Hybrid Transformer-Mamba Language Model"

Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch

Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

Implementation of Diffusion Policy, Toyota Research's supposed breakthrough in leveraging DDPMs for learning policies for real-world Robotics

Implementation of the Large Behavioral Model architecture for dexterous manipulation from Toyota Research Institute

LocoFormer - Generalist Locomotion via Long-Context Adaptation

Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible experimentation and exploration.

Implementation of LVSM, SOTA Large View Synthesis with Minimal 3d Inductive Bias, from Adobe Research

Implementation of AlphaGenome, Deepmind's updated genomic attention model

Implementation of Zorro, Masked Multimodal Transformer, in Pytorch

Implementation of the transformer proposed in "Building Blocks for a Complex-Valued Transformer Architecture"

Implementation of Transframer, Deepmind's U-net + Transformer architecture for up to 30 seconds video generation, in Pytorch

My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)

Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount of time on any token

VAAS is an inference-first, research-driven library for image integrity analysis. It integrates Vision Transformer Attention Mechanisms with patch-level self-consistency analysis to enable fine-grained localization and detection of visual inconsistencies across diverse image analysis tasks.

The open source implementation of the multi grouped query attention by the paper "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"

My implementation of the model KosmosG from "KOSMOS-G: Generating Images in Context with Multimodal Large Language Models"