Repository navigation
Conversation
01275cd to
e09c276
Compare
[Medium impact] The changes affect compiled PyTorch execution for Transformer Engine modules and shared autograd plumbing. The PR needs fixes before merging because recent SummaryAdds full-graph
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Module forward] --> B[Prepare quantizers]
B --> C{Compiled path supported?}
C -- No --> D[Eager forward]
C -- Yes --> E[Fake describes outputs and saved tensors]
E --> F[Custom op runs forward]
F --> G[Return outputs and update weight cache]
G --> H[Backward restores saved tensors]
H --> I[Assign output gradients]
I --> J[Custom op runs backward]
Reviews (40) · Last reviewed commit: "Keep existing Linear behavior in LayerNo..." · Reviewed by Greptile |
Comments Outside DiffThese findings could not be posted inline.
|
Register LayerNorm module forward and backward custom ops with matching fake tensor metadata, shared eligibility checks, and compiled weight caches. Extend the existing compile tests to cover the LayerNorm modules. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
70e4b7d to
83641e8
Compare
|
|
||
| std::optional<size_t> mxfp8_norm_fused_alignment(bool rmsnorm, bool with_2d_quantization, | ||
| bool use_cudnn) { | ||
| if (rmsnorm ? with_2d_quantization : !use_cudnn) { |
There was a problem hiding this comment.
Why is this checking specifically for RMSNorm? This sounds like the original logic had some bug where only RMSNorm was checked rather than any normalization.
There was a problem hiding this comment.
yes, it seems we have but it is hard to obtain it in real training since it is about layernorming 2d scaled tensor (weights) which does not happen. I ll fix in separate PR
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Keep the merged optimize_for_gemm implementation and adapt the LayerNormLinear compile path to the current check_gemm_dims signature. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Restore Linear and fused-attention backward to main. Make normalization gradient setup optional and remove the unused SHAPE field kind and its test. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…-call fast path
… the learning repo
Description
Add full-graph
torch.compilesupport forLayerNormLinearandLayerNormMLP, including backward, dynamic batch dimensions and normalization outputs.The single late fallback check relies on the Dynamo fix in pggPL/pytorch#4, related to pytorch/pytorch#144820. Without that fix, a graph break inside forward's try/finally can disable compilation for subsequent supported instances of the same module class.
Targets
main. This branch includes the NVFP4 packed-buffer, TensorSpec and Bias correctness fixes from NVIDIA/TransformerEngine#3628. Uses the GEMM scale-layout contract merged in NVIDIA/TransformerEngine#3817, so fake outputs follow quantizer metadata without querying native kernel layout rules.Changes