Skip to content

[PyT] Add context-parallel attention benchmark results - #3582

Open
sudhakarsingh27 wants to merge 7 commits into
NVIDIA:mainfrom
sudhakarsingh27:cp_bench
Open

sudhakarsingh27 wants to merge 7 commits into
NVIDIA:mainfrom
sudhakarsingh27:cp_bench

Conversation

@sudhakarsingh27

@sudhakarsingh27 sudhakarsingh27 commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Description

  • Add BF16/FP16 context-parallel benchmark workloads with rank-max endpoint timing and structured JSON output.
  • Add phase-level NVTX ranges and cross-backend checks for profile interpretation and correctness.
  • Publish a concise H100/B200 BF16 THD selection report with seven supporting figures and a direct benchmark/profile example.

The benchmark path deliberately excludes FP8 and CUDA graph changes. Existing correctness and FP8 behavior remain available outside benchmark mode.

Validation

  • Python compilation for all changed Python files
  • Repository license check
  • Black formatting hook
  • Report benchmark instructions and all seven figure links validated
  • Outgoing-content audit for local paths, hostnames, credentials, private issue references, and profiler locations

Distributed THD GPU tests were not rerun on the current validation host because they require sm90 or newer. CI is pending after the latest scope reduction.

Checklist

  • Changes are based on current main
  • Commits are signed off
  • Report documents the measured configurations, method, and limitations
  • No FP8 or CUDA graph benchmark changes are included
  • GPU CI passes

Add reproducible BF16/FP16 CP workloads, mean-of-iteration-rank-max endpoint timing, cross-backend checks, and phase-level NVTX ranges. Keeping benchmark setup and result reduction outside timed samples makes endpoint measurements comparable while preserving the existing correctness and FP8 paths.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Replace the earlier result notes with a concise measured-data report and seven supporting figures. Keeping the publication to a report, snapshots, and a direct benchmark/profile example makes the review self-contained without carrying intermediate analysis tables.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

@sudhakarsingh27 sudhakarsingh27 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Mention TE, cuDNN, pytorch version
  2. Remove the "matching profile"
  3. Talk about second best option as well

@KshitijLakhani KshitijLakhani changed the title Add context-parallel attention benchmark results [PyT] Add context-parallel attention benchmark results Oct 20, 2026
@KshitijLakhani
KshitijLakhani self-requested a review November 2, 2026 19:38
Preserve the independent non-CP reference check introduced upstream while
keeping benchmark-only execution outside the correctness path.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Publish the refreshed measured-data report and its referenced snapshots so
readers can compare both backend latencies without inferring absolute
times from normalized plots. Keep the endpoint values alongside figures
for reproducibility.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@cyanguwa
cyanguwa marked this pull request as ready for review November 18, 2026 18:08
@cyanguwa
cyanguwa self-requested a review as a code owner November 18, 2026 18:08
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the remaining test-discovery gap is non-blocking.

Findings

  1. P2 Comparison tests never reach CI. ▶

Summary

Adds context-parallel attention workloads, rank-max timing, JSON results, and phase-level NVTX ranges.

  • Context-parallel runs can report per-iteration rank-max attention times.
  • Context-parallel attention traces label the work done by each communication phase.
  • The attention report pairs H100 and B200 results with data and figures.

GPU tests were not run during this review.

Reviews (2) · Last reviewed commit: "Keep the context-parallel report focused..." · Reviewed by Greptile

Comment on lines +348 to +351
@pytest.mark.skipif(get_cudnn_version() < (9, 3, 0), reason="cuDNN 9.3.0+ is required.")
@pytest.mark.skipif(get_device_compute_capability() < (9, 0), reason="THD requires sm90+.")
@pytest.mark.parametrize("model", model_configs_cross_backend.keys())
def test_cp_thd_cross_backend_consistency(model):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Comparison tests never reach CI. benchmark_cp.py does not match pytest's default test-file names, and the distributed CI script runs only test_attention_with_cp.py for these attention checks. Importing the workload dictionary does not collect these tests. Add a CI entry point for the smaller cross-method comparisons, while keeping the large stress cases opt-in, so future changes actually run these checks.

Knowledge Base Used: Verification and CI matrix

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread benchmarks/attention/README.md Outdated
@@ -1,222 +1,177 @@
# Choosing Context-Parallel Attention Paths from Measured Data

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. trim .csv file
  2. remove scaling section in README because we're not sure about the conclusion
  3. a moving profile for fused/flash; will update as time goes on

@@ -0,0 +1,426 @@
# Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check if this works well with Alp's PR: #3408

a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True
)
name = a2a_input_names[i - 1]
with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.launch"):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove these sub-ranges. Thanks!

Dense benchmark inputs are allocated per rank, so they bypass the reference path that used to supply the bias chunk indices. Share the dense chunk mapping between both paths and skip indexing broadcast bias. Cover both bias layouts without bringing back full-sequence allocations.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Drop the CP scaling and cross-system ratio sections and their obsolete assets so the report focuses on backend and communication selection. Keep the exact measured latencies and the fields needed to reproduce the figures; document shared settings once in the README.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants