I work on training infra at Periodic Labs.
Previously, I was a tech leader at NVIDIA DevTech, where I built and led the MoE training system in Megatron-LM, enabling efficient training of trillion-parameter MoE models.
Before that, I led training infra at SenseTime and researched distributed training during my master's at Sun Yat-sen University.
The industry's first comprehensive technical report on MoE training infrastructure. Explains how co-optimizing memory, communication, and computation enables DeepSeek-V3-685B training at 1,233 TFLOPS/GPU on NVIDIA GB300.
* Equal contribution. First authors listed alphabetically.
Introduces Parallel Folding, which decouples attention and MoE parallelism so each can be optimized independently. The framework achieves up to 49.3% MFU for Mixtral 8x22B on H100 and scales to 1,024 GPUs.
Contributed to the training infrastructure and performance optimization of the Nemotron 3 model family.
Co-authored a technical blog on MoE training optimizations in Megatron-Core.