data-parallelism

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Making large AI models cheaper, faster and more accessible

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*