Repository navigation
[PyT] Add context-parallel attention benchmark results - #3582
sudhakarsingh27 wants to merge 7 commits into
Conversation
Add reproducible BF16/FP16 CP workloads, mean-of-iteration-rank-max endpoint timing, cross-backend checks, and phase-level NVTX ranges. Keeping benchmark setup and result reduction outside timed samples makes endpoint measurements comparable while preserving the existing correctness and FP8 paths. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Replace the earlier result notes with a concise measured-data report and seven supporting figures. Keeping the publication to a report, snapshots, and a direct benchmark/profile example makes the review self-contained without carrying intermediate analysis tables. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
7da58be to
5d4673d
Compare
sudhakarsingh27
left a comment
There was a problem hiding this comment.
- Mention TE, cuDNN, pytorch version
- Remove the "matching profile"
- Talk about second best option as well
Preserve the independent non-CP reference check introduced upstream while keeping benchmark-only execution outside the correctness path. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Publish the refreshed measured-data report and its referenced snapshots so readers can compare both backend latencies without inferring absolute times from normalized plots. Keep the endpoint values alongside figures for reproducibility. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
[Medium impact] The PR appears safe to merge; the remaining test-discovery gap is non-blocking. Findings
SummaryAdds context-parallel attention workloads, rank-max timing, JSON results, and phase-level NVTX ranges.
GPU tests were not run during this review. Reviews (2) · Last reviewed commit: "Keep the context-parallel report focused..." · Reviewed by Greptile |
| @pytest.mark.skipif(get_cudnn_version() < (9, 3, 0), reason="cuDNN 9.3.0+ is required.") | ||
| @pytest.mark.skipif(get_device_compute_capability() < (9, 0), reason="THD requires sm90+.") | ||
| @pytest.mark.parametrize("model", model_configs_cross_backend.keys()) | ||
| def test_cp_thd_cross_backend_consistency(model): |
There was a problem hiding this comment.
Comparison tests never reach CI.
benchmark_cp.py does not match pytest's default test-file names, and the distributed CI script runs only test_attention_with_cp.py for these attention checks. Importing the workload dictionary does not collect these tests. Add a CI entry point for the smaller cross-method comparisons, while keeping the large stress cases opt-in, so future changes actually run these checks.
Knowledge Base Used: Verification and CI matrix
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| @@ -1,222 +1,177 @@ | |||
| # Choosing Context-Parallel Attention Paths from Measured Data | |||
There was a problem hiding this comment.
- trim .csv file
- remove scaling section in README because we're not sure about the conclusion
- a moving profile for fused/flash; will update as time goes on
| @@ -0,0 +1,426 @@ | |||
| # Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
| a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True | ||
| ) | ||
| name = a2a_input_names[i - 1] | ||
| with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.launch"): |
There was a problem hiding this comment.
Let's remove these sub-ranges. Thanks!
Dense benchmark inputs are allocated per rank, so they bypass the reference path that used to supply the bias chunk indices. Share the dense chunk mapping between both paths and skip indexing broadcast bias. Cover both bias layouts without bringing back full-sequence allocations. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Drop the CP scaling and cross-system ratio sections and their obsolete assets so the report focuses on backend and communication selection. Keep the exact measured latencies and the fields needed to reproduce the figures; document shared settings once in the README. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Description
The benchmark path deliberately excludes FP8 and CUDA graph changes. Existing correctness and FP8 behavior remain available outside benchmark mode.
Validation
Distributed THD GPU tests were not rerun on the current validation host because they require sm90 or newer. CI is pending after the latest scope reduction.
Checklist
main