pretraining

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

Llama中文社区，实时汇总最新Llama学习资料，构建最好的中文Llama大模型开源生态，完全开源可商用

General technology for enabling AI capabilities w/ LLMs and MLLMs

Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

mPLUG-Owl: The Powerful Multi-modal Large Language Model Family

Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).

The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large Models

Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.

PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)

A Chinese Open-Domain Dialogue System

LLMInternSkill: LLM internship resume and job-search Codex Skill for resume polish, JD tailoring, evidence guard, interview grilling, and Project Scout. 大模型实习简历与求职工具箱。

Research code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training"

从 MiniMind 源码读起,再延伸到现代大模型技术体系的中文学习笔记。主线逐行精读预训练 / SFT / DPO / PPO / GRPO 与训练机制;附录 17 篇进阶卷覆盖量化、投机解码、RLHF 全景、模型代际史等 MiniMind 没涉及、但进阶绕不开的主题。

Official codebase for "Next-Latent Prediction Transformers Learn Compact World Models"

PyTorch code for “TVLT: Textless Vision-Language Transformer” (NeurIPS 2022 Oral)

MixGen: A New Multi-Modal Data Augmentation

[ICCV 2025] Explore the Limits of Omni-modal Pretraining at Scale

[NeurIPS 2021] COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

[EMNLP'21] Visual News: Benchmark and Challenges in News Image Captioning

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections. (EMNLP 2022)