attention-mechanism

Implementation of Long-Short Transformer, combining local and global inductive biases for attention over long sequences, in Pytorch

Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems [ICML'25]

Implementation of Infini-Transformer in Pytorch

Implementation of the proposed DeepCrossAttention by Heddes et al at Google research, in Pytorch

A Transformer made of Rotation-equivariant Attention using Vector Neurons

Implementation of Uniformer, a simple attention and 3d convolutional net that achieved SOTA in a number of video classification tasks, debuted in ICLR 2022

Implementation of 🌻 Mirasol, SOTA Multimodal Autoregressive model out of Google Deepmind, in Pytorch

A repository with exploration into using transformers to predict DNA ↔ transcription factor binding

Implementation of Perceiver, General Perception with Iterative Attention

CUDA and Triton implementations of Flash Attention with SoftmaxN.

Implementation of a transformer for reinforcement learning using `x-transformers`

[BMVC 2024] Official repository of the paper titled "MSA^2 Net: Multi-scale Adaptive Attention-guided Network for Medical Image Segmentation"

Code for the paper Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models @ CVPR 2024

Unofficial implementation of GotenNet, new SOTA 3d equivariant transformer, in Pytorch

Implementation of deep implicit attention in PyTorch

[ICLR 2025] Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

Implementation of the world model architecture for self driving out of Wayve

Implementation of OmniNet, Omnidirectional Representations from Transformers, in Pytorch

TrAVis: Visualise BERT attention in your browser

Exploration into the proposed "Self Reasoning Tokens" by Felipe Bonetto