attention-mechanism

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

邱锡鹏《神经网络与深度学习》（蒲公英书）理论书 v2 与通识版

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

A concise but complete full-attention transformer with a set of promising experimental features from various papers

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites

RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

Keras Attention Layer (Luong and Bahdanau scores).

Build a modern LLM from scratch. Every line commented. Explained like we are five.

My implementation of the original GAT paper (Veličković et al.). I've additionally included the playground.py file for visualizing the Cora dataset, GAT embeddings, an attention mechanism, and entropy histograms. I've supported both Cora (transductive) and PPI (inductive) examples!

Reformer, the efficient Transformer, in Pytorch

Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch

Learn LLM internals step by step - from tokenization to attention to inference optimization.

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

A Deep Learning library for EEG Tasks (Signals) Classification, based on TensorFlow.