transformers

A JAX Research Toolkit for Visualizing, Manipulating, and Understanding Gemma Models with Multi-modal Support based on Penzai.

Implementation of Zorro, Masked Multimodal Transformer, in Pytorch

[AAAI 2022] This is the official PyTorch implementation of "Less is More: Pay Less Attention in Vision Transformers"

Efficient Attention for Long Sequence Processing

code and data to faciliate BERT/ELECTRA for document ranking. Details refer to the paper - PARADE: Passage Representation Aggregation for Document Reranking.

解锁HuggingFace生态的百般用法

A collection of resources on using BERT (https://arxiv.org/abs/1810.04805 ) and related Language Models in production environments.

A mental-model-first field guide to Audio AI — from raw waveforms to production voice systems. Covers ASR, TTS, audio classification, Transformer architectures (CTC/Seq2Seq), fine-tuning, and real-world pipelines like voice assistants and meeting transcription. Bilingual (EN/FA).

A minimal TensorFlow.js re-implementation of Karpathy's minGPT (Generative Pre-trained Transformer). The GPT model itself is <300 lines of code.

🙈 An unofficial implementation of SoftMaskedBert based on huggingface/transformers.

SAINT PyTorch implementation

MCore-Bridge: Providing Megatron-Core model definitions for state-of-the-art large models and making Megatron training as simple as Transformers — with support for 300+ large language models (Qwen3-Next, GLM-5.2, Deepseek-V4, MiniMax-2.7, ...) and 200+ multimodal large models (Qwen3.5, Qwen3-Omni, Gemma4, ...).

Open source subtitling platform 💻 for transcribing and translating videos/audios in Indic languages.

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Dual Diffusion is a generative diffusion model for music trained on video game soundtracks.

Granite Switch — Build AI models like you build software

Implementation of the transformer proposed in "Building Blocks for a Complex-Valued Transformer Architecture"

Implementation of 🌻 Mirasol, SOTA Multimodal Autoregressive model out of Google Deepmind, in Pytorch

[ICLR 2023] "Learning to Grow Pretrained Models for Efficient Transformer Training" by Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Cox, Zhangyang Wang, Yoon Kim

Slovo: Russian Sign Language Dataset and Models