attention-mechanism

A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT, ChatGPT, LLMs, and transfer learning.

My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways

Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification

Implementation of Bottleneck Transformer in Pytorch

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

FlashAttention (Metal Port)

Build high-performance AI models with modular building blocks

Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch

Implementation of Parti, Google's pure attention-based text-to-image neural network, in Pytorch

A novel implementation of fusing ViT with Mamba into a fast, agile, and high performance Multi-Modal Model. Powered by Zeta, the simplest AI framework ever.

Open-source pre-training implementation of Google's LaMDA in PyTorch. Adding RLHF similar to ChatGPT.

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

Python library & framework to build custom translators for the hearing-impaired and translate between Sign Language & Text using Artificial Intelligence.

Implementation of "PaLM-E: An Embodied Multimodal Language Model"

End-to-End speech recognition implementation base on TensorFlow (CTC, Attention, and MTL training)

PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (https://arxiv.org/abs/2404.07143)

Implementation of a Transformer, but completely in Triton

My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"