xrsrke/pipegoose

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

⭐ 87 ⑂ 20 Python MIT · 2023-12-14推送

data-parallelism huggingface-transformers megatron mixture-of-experts model-parallelism moe pipeline-parallelism transformers tensor-parallelism megatron-lm 3d-parallelism distributed-optimizers large-scale-language-modeling sequence-parallelism zero-1

87

Watchers

0

贡献者

0

Commits

0

Releases

32

Open Issues

2023-12-14

最近推送

原文中文

暂无 README