tensor-parallelism

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

A distributed training framework for large language models powered by Lightning.

DeepSeek-V4-Flash-0731 tuned on 4x DGX Spark (GB10) at TP=4 — 123.13 tok/s decode. Every variable measured one at a time, with the values that lost.

Tencent Hy3 295B MoE (NVFP4) on 2x NVIDIA DGX Spark — TP2 over 200GbE, 256K context, 26 tok/s end-to-end. Tuned, benchmarked, agent-ready.