You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Enabling sequence_parallel slows down training with fp16 #182
I am testing GPT2 model training using TransformerLayer.
Training slows down significantly when sequence_parallel=True, achieves 1/5th of throughput of training without sequence_parallel.
I also observe that sequence_parallel=True results in OOM for some batch sizes where sequence_parallel=False can run successfully.
Do you have any recommendation to achieve better throughput with sequence_parallel and fp16?
Model is ~4.3B with 12 layers, tp_size=4, fp16, seq_len=2048, training with 8 A100 GPUs.
Which toolkit (e.g. NeMo) are you using to enable sequence parallelism? Does that toolkit support sequence parallel? Simply passing sequence_parallel=True in TE will not do the trick. The input during the forward pass must also be split in the sequence dimension among the tensor parallel workers before passing it to TE's TransformerLayer.
If this is the case, an unsharded input will be gathered across the TP group during fprop, making the input larger and explaining the extra memory consumption as well as slowdown observed. I think this is what is happening here, does that make sense?
Activity
ptrendx commentedon Apr 28, 2023
@ksivaman Could you take a look?
rahul003 commentedon May 9, 2023
Hey @ptrendx and @ksivaman Any update on this?
ksivaman commentedon May 9, 2023
Which toolkit (e.g. NeMo) are you using to enable sequence parallelism? Does that toolkit support sequence parallel? Simply passing
sequence_parallel=Truein TE will not do the trick. The input during the forward pass must also be split in the sequence dimension among the tensor parallel workers before passing it to TE'sTransformerLayer.If this is the case, an
unshardedinput will be gathered across the TP group during fprop, making the input larger and explaining the extra memory consumption as well as slowdown observed. I think this is what is happening here, does that make sense?ksivaman commentedon May 9, 2023
Could you confirm if this is the issue? Either way, I will make a change soon that catches this behavior and reports an error.
rahul003 commentedon Jun 5, 2023
Yes thanks for the help