diff --git a/benchmarks/attention/README.md b/benchmarks/attention/README.md new file mode 100644 index 00000000000..0fc72a4e9fd --- /dev/null +++ b/benchmarks/attention/README.md @@ -0,0 +1,136 @@ +# Choosing Faster Context-Parallel Attention Paths + +Choosing a fast context-parallel (CP) attention configuration means selecting +the backend, communication method, and CP size together. These H100 and B200 +benchmarks show how workload shape and attention pattern guide that choice, +using complete forward-plus-backward attention latency as the objective. + +## What we measured + +All figures derive from forward-plus-backward attention latency; the captions +define each comparison and ratio. + +| Dimension | Configuration | +| --- | --- | +| Inputs | BF16 packed THD; 32 query / 8 key-value heads; head dimension 128; no dropout or gaps | +| Workload | `B*S=524288` tokens; uniform sequences, with batch size adjusted to keep tokens fixed | +| Causal sweep | Nine shapes: `1x512k` through `256x2k`, halving sequence length each step; `k=1024` | +| Sliding-window attention (SWA) | First five shapes, through `16x32k`; causal window `(1024,0)` | +| Parallelism | CP-only eager execution on 2, 4, or 8 GPUs; P2P, all-gather, A2A; SWA excludes P2P | +| Timing | 10 warmups, then 50 iterations; mean of per-iteration rank-maximum latency; no CUDA graphs | + +The [444 endpoint values](results/context_parallel/endpoints_20261116.csv) +use TE `5d4673da00226be3772cdb135db4cdd0e7b8edeb`, CUDA 13.4, cuDNN 9.27/frontend +1.30, NCCL 2.32.3, and PyTorch 2.15 development builds. Fused means cuDNN +FusedAttention. Flash is FA3 on H100; B200 uses FA4 b11/DSL 4.4.2 for causal +and b33/DSL 4.8 for SWA. These results describe the listed versions; +remeasure backend choices when the software changes. + +## Start with the actual shape + +At fixed token count, shorter sequences reduce the leading dense-attention +work term, `B*S²`. Latency generally falls; several P2P and all-gather curves +flatten at short sequences especially when CP Size >= 4, which is due to +shrinking attention work that exposes communication costs. +Refer to this graph to find out the general workload size and their expected +performance across backend, communication types and GPUs. + +![Causal endpoint sweep with Fused and Flash in separate rows](results/context_parallel/images/cp_causal_endpoint_sweep_by_backend_review_v2.webp) + +*Figure 1. Absolute causal latency. Sequence length decreases left to right; +both axes are logarithmic. Color identifies hardware, line style identifies +communication. The Flash row changes both GPU and implementation: H100 FA3 +versus B200 FA4. Colors are local to each figure's legend.* + +## Choose backend and communication together + +First ask which backend is faster within a chosen communication method. +In Figure 2, rows are workload configs and each column fixes both CP size and method. +Cell colors show the direction and size of the latency difference. + +![H100 causal latency pairs, Fused / FA3 in milliseconds](results/context_parallel/images/cp_backend_latencies_h100_review_v1.webp) + +![B200 causal latency pairs, Fused / FA4 in milliseconds](results/context_parallel/images/cp_backend_latencies_b200_review_v1.webp) + +*Figure 2. Cells show Fused / Flash latency in ms at fixed GPU, shape, CP, +and communication method; bold marks the lower value. Green favors Fused, +blue favors Flash. Both panels share a gradient for +`100*(slower-faster)/slower`, calculated before rounding.* + +For H100 CP8 `256x2k`, Fused is faster than FA3 when both use P2P; +FA3 is faster than Fused when both use A2A. +The preferred backend changes with the communication method. Figure 3 asks +the complementary question: which method is fastest for each backend? + +![H100 context-parallel attention latency relative to the fastest method](results/context_parallel/images/cp_communication_dotplot_h100_review_v4.webp) + +![B200 context-parallel attention latency relative to the fastest method](results/context_parallel/images/cp_communication_dotplot_b200_review_v4.webp) + +*Figure 3. Each dot is attention latency divided by the fastest method at +fixed GPU/backend/shape/CP; 1× is best. Row labels give that baseline in ms. +CP columns use different x-axis ranges; use numeric ratios for comparisons +across columns. Compare the printed baseline milliseconds for the same GPU +and shape to select among backends or CP sizes.* + +In that H100 example, A2A is fastest for both backends, and FA3 has the lower +A2A latency. Combining the comparisons selects FA3/A2A. Across the measured +causal grid, FA3 supplies the best joint choice for every H100 shape/CP tuple. +B200's Fused joint wins occur at `256x2k`; its CP4/CP8 A2A differences are small. + +A2A is a strong first candidate at causal CP4/CP8, especially for short +shapes. At CP2, method ordering changes with shape, backend, and hardware. +Use these patterns to prioritize the supported combinations to benchmark. +If the fastest method is unavailable, Figure 3 shows the next supported option +and its latency penalty at the same GPU/backend/shape/CP. + +The benchmarked pure-A2A path shards heads: CP size must divide both local +query and key/value head counts (after any tensor parallelism), so our +32-query/8-KV-head setup caps this path at CP8. + +## Choose communication for sliding-window attention + +With a fixed window and token count, leading attention work is approximately +constant across these shapes. Most SWA paths are nearly flat; H100 Fused +all-gather is the exception: latency grows strongly with +sequence length, while its A2A path stays nearly flat. Compare communication +methods within each hardware/backend panel to build the SWA shortlist. + +![SWA endpoints for all four hardware and backend scopes](results/context_parallel/images/cp_swa_comm_clear_labels_review_v3.webp) + +*Figure 4. Orange/open markers denote all-gather; purple/filled markers denote +A2A. Line styles and marker shapes identify CP size. Sequence length increases +left to right. Each panel has its own logarithmic y scale; use its labeled +milliseconds when reading latency.* + +At SWA CP2, all-gather is lower for H100 FA3 and both B200 backends; H100 +Fused instead favors A2A. At CP4/CP8, A2A is lower in all four scopes. + +## A small selection loop + +1. Fix the workload and target environment: mask/window, shape, precision, + layout, heads, GPU, and software versions. +2. For each feasible CP size, compare supported **backend × communication** + combinations, using Figures 2–3 and 4 to prioritize candidates. +3. Choose by absolute forward-plus-backward latency. Then evaluate that + configuration in the full training workload and resource budget. +4. Verify outputs and gradients; repeat close comparisons. Benchmark additional + configurations directly on the target environment. + +## Run one benchmark + +This runs the `64x8k` BF16 THD training case with FusedAttention, CP8 and A2A: + +```bash +NVTE_CP_BENCH_ONLY=1 NVTE_NVTX_ENABLED=0 \ +torchrun --standalone --nproc-per-node=8 \ + tests/pytorch/attention/run_attention_with_cp.py \ + dtype=bf16 model=uniform_64x8k qkv_format=thd \ + kernel_backend=FusedAttention cp_comm_type=a2a \ + thd_seqlen_pattern=max benchmark=50 log_level=WARNING +``` + +`CP_BENCH_RESULT` reports per-iteration rank maxima and their mean. Input +construction, synchronization before timing, and result reduction are excluded. +Aliases are in `tests/pytorch/attention/benchmark_cp.py`; benchmark mode here +supports FP16/BF16. See +[Transformer Engine PR #3582](https://github.com/NVIDIA/TransformerEngine/pull/3582). diff --git a/benchmarks/attention/results/context_parallel/endpoints_20261116.csv b/benchmarks/attention/results/context_parallel/endpoints_20261116.csv new file mode 100644 index 00000000000..eb97d24c201 --- /dev/null +++ b/benchmarks/attention/results/context_parallel/endpoints_20261116.csv @@ -0,0 +1,445 @@ +hardware,backend,batch_size,sequence_length,cp_size,communication_type,latency_ms,workload,backend_version +B200,FA4,256,2048,2,a2a,48.054356812499464,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,2,all_gather,52.444414952769876,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,2,p2p,43.950961725786328,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,4,a2a,25.563965234905481,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,4,all_gather,43.545562126673758,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,4,p2p,41.580961989238858,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,8,a2a,13.898696980904788,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,8,all_gather,39.332058555446565,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,256,2048,8,p2p,43.69566376786679,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,2,a2a,58.875222466886044,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,2,all_gather,58.632435807958245,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,2,p2p,53.644937453791499,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,4,a2a,30.933826328255236,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,4,all_gather,43.476167684420943,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,4,p2p,41.534842294640839,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,8,a2a,15.918515305966139,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,8,all_gather,37.109900815412402,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,128,4096,8,p2p,40.628582523204386,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,2,a2a,83.861514702439308,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,2,all_gather,81.805726550519466,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,2,p2p,76.547574438154697,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,4,a2a,43.915478000417352,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,4,all_gather,53.139505973085761,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,4,p2p,51.803192859515548,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,8,a2a,22.08871562499553,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,8,all_gather,39.770964281633496,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,64,8192,8,p2p,42.849678010679781,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,2,a2a,133.18054937757552,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,2,all_gather,130.5657961871475,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,2,p2p,124.49247082695365,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,4,a2a,68.732839580625296,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,4,all_gather,78.422334381612018,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,4,p2p,75.135979540646076,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,8,a2a,35.191969661973417,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,8,all_gather,50.57233514264226,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,32,16384,8,p2p,52.86088886205107,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,2,a2a,230.4513970669359,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,2,all_gather,231.59929948393255,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,2,p2p,223.31589743960649,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,4,a2a,118.7615633290261,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,4,all_gather,127.69025882706046,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,4,p2p,124.40667238086462,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,8,a2a,59.756720825098455,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,8,all_gather,75.930685950443149,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,16,32768,8,p2p,76.300307223573327,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,2,a2a,430.92415824066848,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,2,all_gather,428.62442838959396,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,2,p2p,416.64519423618913,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,4,a2a,216.46104000508785,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,4,all_gather,228.60902026295662,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,4,p2p,219.84899477101862,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,8,a2a,108.82024500053376,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,8,all_gather,126.65563381277025,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,8,65536,8,p2p,123.71700961142778,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,2,a2a,827.99675359856337,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,2,all_gather,819.29060115944594,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,2,p2p,799.00695895310491,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,4,a2a,412.63314326293766,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,4,all_gather,426.41620861366391,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,4,p2p,413.79107009619474,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,8,a2a,206.89831985160708,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,8,all_gather,228.63264725543559,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,4,131072,8,p2p,219.63677287101746,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,2,a2a,1622.0335530582815,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,2,all_gather,1603.3858635881916,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,2,p2p,1604.5129761239514,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,4,a2a,799.77706050965935,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,4,all_gather,827.32836582697928,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,4,p2p,803.89650798868388,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,8,a2a,402.80663092620671,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,8,all_gather,426.39894501306117,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,2,262144,8,p2p,407.71242534741759,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,2,a2a,3211.4910259982571,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,2,all_gather,3176.5411229757592,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,2,p2p,3182.307100542821,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,4,a2a,1579.5671898173168,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,4,all_gather,1619.8011543089524,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,4,p2p,1586.3398853782564,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,8,a2a,799.60970754735172,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,8,all_gather,825.04722489509732,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FA4,1,524288,8,p2p,779.38508310006,causal,FA4 4.0.0b11 / DSL 4.4.2 +B200,FusedAttention,256,2048,2,a2a,48.908770605921745,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,2,all_gather,41.083853002637625,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,2,p2p,41.564601697027683,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,4,a2a,25.520598203875124,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,4,all_gather,29.701541070826352,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,4,p2p,32.708454444073141,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,8,a2a,13.748052964219823,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,8,all_gather,26.149557423777878,causal,cuDNN 9.27 +B200,FusedAttention,256,2048,8,p2p,33.997357529588044,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,2,a2a,63.791222218424082,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,2,all_gather,56.162430867552757,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,2,p2p,57.296145875006914,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,4,a2a,32.258549220860004,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,4,all_gather,37.241728650406003,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,4,p2p,40.032183835282922,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,8,a2a,16.642578449100256,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,8,all_gather,27.920442190952599,causal,cuDNN 9.27 +B200,FusedAttention,128,4096,8,p2p,34.978153919801116,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,2,a2a,94.372578132897615,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,2,all_gather,86.351215336471796,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,2,p2p,87.663502357900143,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,4,a2a,47.58431414142251,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,4,all_gather,52.435254212468863,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,4,p2p,55.811340985819697,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,8,a2a,23.535264418460429,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,8,all_gather,35.263232779689133,causal,cuDNN 9.27 +B200,FusedAttention,64,8192,8,p2p,40.896539581008255,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,2,a2a,152.42091465741396,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,2,all_gather,147.76612211950123,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,2,p2p,153.78411630168557,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,4,a2a,76.253210883587599,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,4,all_gather,83.264878340996802,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,4,p2p,87.243960490450263,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,8,a2a,38.391461605206132,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,8,all_gather,50.537892980501056,causal,cuDNN 9.27 +B200,FusedAttention,32,16384,8,p2p,56.380011686123908,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,2,a2a,266.32862573023885,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,2,all_gather,269.83803644310683,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,2,p2p,292.21589621622115,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,4,a2a,132.29394142515957,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,4,all_gather,143.682677205652,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,4,p2p,152.48120967298746,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,8,a2a,65.5210971692577,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,8,all_gather,81.175999869592488,causal,cuDNN 9.27 +B200,FusedAttention,16,32768,8,p2p,87.731654057279229,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,2,a2a,498.90749248210341,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,2,all_gather,513.44526209402829,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,2,p2p,558.19858033210039,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,4,a2a,243.81056954152882,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,4,all_gather,265.13517287559807,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,4,p2p,285.61297591775656,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,8,a2a,119.57734327297658,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,8,all_gather,142.12203004863113,causal,cuDNN 9.27 +B200,FusedAttention,8,65536,8,p2p,151.48329581134021,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,2,a2a,963.00193553324789,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,2,all_gather,999.92517116479576,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,2,p2p,1098.6345439963043,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,4,a2a,469.68333456665277,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,4,all_gather,509.24768427852541,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,4,p2p,543.28304182738066,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,8,a2a,230.16856563277543,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,8,all_gather,265.48301762901247,causal,cuDNN 9.27 +B200,FusedAttention,4,131072,8,p2p,279.67756920494139,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,2,a2a,1886.3898558914661,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,2,all_gather,1984.2398227704689,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,2,p2p,2204.5954379998147,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,4,a2a,915.2850160561502,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,4,all_gather,997.58198171388358,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,4,p2p,1077.4534165905789,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,8,a2a,449.88746404647827,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,8,all_gather,511.50170382112265,causal,cuDNN 9.27 +B200,FusedAttention,2,262144,8,p2p,530.46104622539133,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,2,a2a,3761.8458892730996,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,2,all_gather,4024.9472429789603,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,2,p2p,4408.9846629602835,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,4,a2a,1820.5031955288723,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,4,all_gather,1986.7245514690876,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,4,p2p,2105.3235159954056,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,8,a2a,895.90696591883898,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,8,all_gather,1003.0214164592326,causal,cuDNN 9.27 +B200,FusedAttention,1,524288,8,p2p,1053.293392380001,causal,cuDNN 9.27 +H100,FA3,256,2048,2,a2a,72.054390422999859,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,2,all_gather,76.6713578812778,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,2,p2p,66.934799095615745,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,4,a2a,40.764777222648263,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,4,all_gather,61.35681614279747,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,4,p2p,63.276550555601716,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,8,a2a,22.581347273662686,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,8,all_gather,53.65687089972198,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,256,2048,8,p2p,62.903919629752636,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,2,a2a,95.720105795189738,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,2,all_gather,97.520042834803462,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,2,p2p,89.632083727046847,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,4,a2a,51.496356092393398,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,4,all_gather,70.889552300795913,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,4,p2p,69.671720033511519,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,8,a2a,27.749658841639757,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,8,all_gather,58.205432072281837,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,128,4096,8,p2p,66.553244506940246,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,2,a2a,143.71846800670028,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,2,all_gather,144.23281536437571,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,2,p2p,138.1003799289465,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,4,a2a,75.418664114549756,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,4,all_gather,92.739104824140668,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,4,p2p,92.17351695522666,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,8,a2a,39.612147370353341,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,8,all_gather,68.527591777965426,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,64,8192,8,p2p,72.779268277809024,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,2,a2a,239.68915583565831,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,2,all_gather,240.21987156942487,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,2,p2p,235.79713087528944,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,4,a2a,124.17133287526667,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,4,all_gather,140.61497073620558,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,4,p2p,141.60285881720483,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,8,a2a,64.061724850907922,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,8,all_gather,89.875792674720287,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,32,16384,8,p2p,93.309187721461058,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,2,a2a,432.51852962188423,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,2,all_gather,431.64120366796851,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,2,p2p,429.3683760240674,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,4,a2a,221.5066453628242,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,4,all_gather,237.42138759233057,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,4,p2p,241.94990732707083,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,8,a2a,112.93300336226821,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,8,all_gather,139.19929572381079,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,16,32768,8,p2p,141.8358863517642,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,2,a2a,818.32612715661526,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,2,all_gather,820.46166927553713,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,2,p2p,816.59245923161507,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,4,a2a,415.78726929612458,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,4,all_gather,432.21457412466407,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,4,p2p,444.10763795487583,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,8,a2a,210.11575753800571,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,8,all_gather,236.23404284007847,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,8,65536,8,p2p,240.19099005497992,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,2,a2a,1588.5138618946075,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,2,all_gather,1597.956011723727,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,2,p2p,1604.2189317755401,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,4,a2a,803.32782062701881,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,4,all_gather,823.93877189606428,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,4,p2p,842.77348931878805,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,8,a2a,404.38399104401469,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,8,all_gather,430.61719696968794,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,4,131072,8,p2p,439.3602732848376,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,2,a2a,3128.5032724589109,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,2,all_gather,3149.1661213710904,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,2,p2p,3176.3285571243614,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,4,a2a,1578.0215575266629,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,4,all_gather,1606.4557362627238,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,4,p2p,1650.8648582454771,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,8,a2a,791.35394369252026,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,8,all_gather,821.98780873790383,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,2,262144,8,p2p,833.62059248611331,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,2,a2a,6217.7503936644644,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,2,all_gather,6221.2544392794371,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,2,p2p,6319.3347298447043,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,4,a2a,3105.7554217521101,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,4,all_gather,3168.6095122154802,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,4,p2p,3289.7820385824889,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,8,a2a,1565.4683590028435,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,8,all_gather,1603.0500869546086,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FA3,1,524288,8,p2p,1623.2887550909072,causal,FA3 source 94e22c906678e5483fa0e9e24d8e787bc2c0ed4c +H100,FusedAttention,256,2048,2,a2a,77.784911999478936,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,2,all_gather,76.876676995307207,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,2,p2p,71.044913846999407,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,4,a2a,43.066594284027815,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,4,all_gather,55.914713386446238,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,4,p2p,61.712074754759669,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,8,a2a,23.887596409767866,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,8,all_gather,47.911794735118747,causal,cuDNN 9.27 +H100,FusedAttention,256,2048,8,p2p,59.764790656045079,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,2,a2a,105.76797412708402,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,2,all_gather,108.09721384197474,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,2,p2p,98.978356933221221,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,4,a2a,56.434218175709248,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,4,all_gather,68.775189202278852,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,4,p2p,72.649620259180665,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,8,a2a,29.969938835129142,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,8,all_gather,52.841963395476341,causal,cuDNN 9.27 +H100,FusedAttention,128,4096,8,p2p,62.901229709386826,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,2,a2a,163.11693766154349,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,2,all_gather,173.52889441885054,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,2,p2p,156.08295458368957,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,4,a2a,84.883466102182865,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,4,all_gather,97.989189764484763,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,4,p2p,101.50442107580602,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,8,a2a,44.096193751320243,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,8,all_gather,64.976866897195578,causal,cuDNN 9.27 +H100,FusedAttention,64,8192,8,p2p,73.68305972777307,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,2,a2a,277.08275532349944,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,2,all_gather,303.62422602251172,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,2,p2p,269.33990127407014,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,4,a2a,142.05479382537305,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,4,all_gather,158.94928198307753,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,4,p2p,162.74472334422171,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,8,a2a,72.894939333200455,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,8,all_gather,93.19234624505043,causal,cuDNN 9.27 +H100,FusedAttention,32,16384,8,p2p,101.18579210713506,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,2,a2a,503.40858266688883,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,2,all_gather,564.60915856994689,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,2,p2p,494.40276429057121,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,4,a2a,255.34638489596546,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,4,all_gather,279.40872807055712,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,4,p2p,285.77299722470343,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,8,a2a,129.41237631253898,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,8,all_gather,152.35139810480177,causal,cuDNN 9.27 +H100,FusedAttention,16,32768,8,p2p,160.38432019762695,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,2,a2a,961.50172301568091,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,2,all_gather,1094.4265370164067,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,2,p2p,936.00315142422915,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,4,a2a,482.41237071342766,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,4,all_gather,525.15446201898158,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,4,p2p,530.416398961097,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,8,a2a,243.42559181153774,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,8,all_gather,269.96284984052181,causal,cuDNN 9.27 +H100,FusedAttention,8,65536,8,p2p,280.85014368407428,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,2,a2a,1920.2506162133068,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,2,all_gather,2193.7368546146899,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,2,p2p,1837.920605391264,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,4,a2a,962.29234612546861,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,4,all_gather,1035.461913049221,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,4,p2p,1025.2504296507686,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,8,a2a,485.01088473014534,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,8,all_gather,513.23397384956479,causal,cuDNN 9.27 +H100,FusedAttention,4,131072,8,p2p,523.34081172011793,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,2,a2a,3860.2947339322418,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,2,all_gather,4417.9239407088608,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,2,p2p,3720.4797976650298,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,4,a2a,1926.1512585636228,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,4,all_gather,2055.5587933585048,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,4,p2p,1962.9440429247916,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,8,a2a,970.92637352645397,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,8,all_gather,1008.369969567284,causal,cuDNN 9.27 +H100,FusedAttention,2,262144,8,p2p,1002.7833829820156,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,2,a2a,7742.2557545360178,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,2,all_gather,8819.2213711049408,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,2,p2p,7576.812039501965,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,4,a2a,3852.9599700588733,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,4,all_gather,4095.3519247751683,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,4,p2p,3850.1774093136191,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,8,a2a,1932.9932362679392,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,8,all_gather,2000.4731373302639,causal,cuDNN 9.27 +H100,FusedAttention,1,524288,8,p2p,1967.4360298924148,causal,cuDNN 9.27 +B200,FA4,16,32768,2,a2a,47.756819198839366,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,16,32768,2,all_gather,42.116498877294362,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,16,32768,4,a2a,25.112267241347581,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,16,32768,4,all_gather,32.698509141337126,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,16,32768,8,a2a,13.538068009074777,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,16,32768,8,all_gather,28.418416224885732,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,8,65536,2,a2a,48.322233601938933,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,8,65536,2,all_gather,42.163606067188084,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,8,65536,4,a2a,25.47616406576708,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,8,65536,4,all_gather,32.613212238065898,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,8,65536,8,a2a,13.641242596786469,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,8,65536,8,all_gather,28.248160120565444,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,4,131072,2,a2a,48.610414597205818,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,4,131072,2,all_gather,42.255942304618657,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,4,131072,4,a2a,25.396472811698914,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,4,131072,4,all_gather,32.823026443365961,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,4,131072,8,a2a,13.609639981295913,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,4,131072,8,all_gather,28.091348176822066,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,2,262144,2,a2a,48.653520259540528,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,2,262144,2,all_gather,42.67089941771701,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,2,262144,4,a2a,25.497894915752113,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,2,262144,4,all_gather,32.870800986420363,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,2,262144,8,a2a,13.498624647036195,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,2,262144,8,all_gather,28.145790100097656,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,1,524288,2,a2a,48.51829192833975,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,1,524288,2,all_gather,42.64432662865147,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,1,524288,4,a2a,25.522072752937675,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,1,524288,4,all_gather,32.809982569888234,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,1,524288,8,a2a,13.542475909925997,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FA4,1,524288,8,all_gather,28.069369662553072,swa1024,FA4 4.0.0b33 / DSL 4.8.0 +B200,FusedAttention,16,32768,2,a2a,50.880736539838836,swa1024,cuDNN 9.27.0 +B200,FusedAttention,16,32768,2,all_gather,42.279383997083642,swa1024,cuDNN 9.27.0 +B200,FusedAttention,16,32768,4,a2a,26.386046061525121,swa1024,cuDNN 9.27.0 +B200,FusedAttention,16,32768,4,all_gather,30.314159781555645,swa1024,cuDNN 9.27.0 +B200,FusedAttention,16,32768,8,a2a,13.887069899938069,swa1024,cuDNN 9.27.0 +B200,FusedAttention,16,32768,8,all_gather,25.114756780094467,swa1024,cuDNN 9.27.0 +B200,FusedAttention,8,65536,2,a2a,50.744277319172397,swa1024,cuDNN 9.27.0 +B200,FusedAttention,8,65536,2,all_gather,42.472477339906618,swa1024,cuDNN 9.27.0 +B200,FusedAttention,8,65536,4,a2a,26.296818503178656,swa1024,cuDNN 9.27.0 +B200,FusedAttention,8,65536,4,all_gather,30.208658217452466,swa1024,cuDNN 9.27.0 +B200,FusedAttention,8,65536,8,a2a,13.916273798677139,swa1024,cuDNN 9.27.0 +B200,FusedAttention,8,65536,8,all_gather,24.977692060638219,swa1024,cuDNN 9.27.0 +B200,FusedAttention,4,131072,2,a2a,51.013421101961285,swa1024,cuDNN 9.27.0 +B200,FusedAttention,4,131072,2,all_gather,42.676416759495623,swa1024,cuDNN 9.27.0 +B200,FusedAttention,4,131072,4,a2a,26.402450522291474,swa1024,cuDNN 9.27.0 +B200,FusedAttention,4,131072,4,all_gather,30.427166937151924,swa1024,cuDNN 9.27.0 +B200,FusedAttention,4,131072,8,a2a,13.979041782440618,swa1024,cuDNN 9.27.0 +B200,FusedAttention,4,131072,8,all_gather,25.11584586231038,swa1024,cuDNN 9.27.0 +B200,FusedAttention,2,262144,2,a2a,50.732666358235292,swa1024,cuDNN 9.27.0 +B200,FusedAttention,2,262144,2,all_gather,42.872044339310378,swa1024,cuDNN 9.27.0 +B200,FusedAttention,2,262144,4,a2a,26.629689518595114,swa1024,cuDNN 9.27.0 +B200,FusedAttention,2,262144,4,all_gather,30.470655396929942,swa1024,cuDNN 9.27.0 +B200,FusedAttention,2,262144,8,a2a,13.946448779315688,swa1024,cuDNN 9.27.0 +B200,FusedAttention,2,262144,8,all_gather,25.157409141538665,swa1024,cuDNN 9.27.0 +B200,FusedAttention,1,524288,2,a2a,50.964459800743498,swa1024,cuDNN 9.27.0 +B200,FusedAttention,1,524288,2,all_gather,42.908197800279595,swa1024,cuDNN 9.27.0 +B200,FusedAttention,1,524288,4,a2a,26.378448121249676,swa1024,cuDNN 9.27.0 +B200,FusedAttention,1,524288,4,all_gather,30.259305580984801,swa1024,cuDNN 9.27.0 +B200,FusedAttention,1,524288,8,a2a,13.896055340301245,swa1024,cuDNN 9.27.0 +B200,FusedAttention,1,524288,8,all_gather,25.020858541247435,swa1024,cuDNN 9.27.0 +H100,FA3,16,32768,2,a2a,75.399251555791125,swa1024,FA3 3.0.0 +H100,FA3,16,32768,2,all_gather,70.363729135133326,swa1024,FA3 3.0.0 +H100,FA3,16,32768,4,a2a,41.863747952738777,swa1024,FA3 3.0.0 +H100,FA3,16,32768,4,all_gather,52.149094180203974,swa1024,FA3 3.0.0 +H100,FA3,16,32768,8,a2a,23.120873779989779,swa1024,FA3 3.0.0 +H100,FA3,16,32768,8,all_gather,43.662725517060608,swa1024,FA3 3.0.0 +H100,FA3,8,65536,2,a2a,75.804170710034668,swa1024,FA3 3.0.0 +H100,FA3,8,65536,2,all_gather,73.236498741898686,swa1024,FA3 3.0.0 +H100,FA3,8,65536,4,a2a,42.082877220818773,swa1024,FA3 3.0.0 +H100,FA3,8,65536,4,all_gather,52.643666256917641,swa1024,FA3 3.0.0 +H100,FA3,8,65536,8,a2a,23.232433499069884,swa1024,FA3 3.0.0 +H100,FA3,8,65536,8,all_gather,43.559533858206123,swa1024,FA3 3.0.0 +H100,FA3,4,131072,2,a2a,76.859887704486027,swa1024,FA3 3.0.0 +H100,FA3,4,131072,2,all_gather,73.857643257360905,swa1024,FA3 3.0.0 +H100,FA3,4,131072,4,a2a,42.485602044034749,swa1024,FA3 3.0.0 +H100,FA3,4,131072,4,all_gather,53.966472045285627,swa1024,FA3 3.0.0 +H100,FA3,4,131072,8,a2a,23.23970235651359,swa1024,FA3 3.0.0 +H100,FA3,4,131072,8,all_gather,43.776596641400829,swa1024,FA3 3.0.0 +H100,FA3,2,262144,2,a2a,76.481266018236056,swa1024,FA3 3.0.0 +H100,FA3,2,262144,2,all_gather,73.176326880929992,swa1024,FA3 3.0.0 +H100,FA3,2,262144,4,a2a,42.098464962327853,swa1024,FA3 3.0.0 +H100,FA3,2,262144,4,all_gather,54.515684702200815,swa1024,FA3 3.0.0 +H100,FA3,2,262144,8,a2a,23.319632903439924,swa1024,FA3 3.0.0 +H100,FA3,2,262144,8,all_gather,44.339008742244914,swa1024,FA3 3.0.0 +H100,FA3,1,524288,2,a2a,76.661934575531632,swa1024,FA3 3.0.0 +H100,FA3,1,524288,2,all_gather,72.754019002895802,swa1024,FA3 3.0.0 +H100,FA3,1,524288,4,a2a,42.182049901457503,swa1024,FA3 3.0.0 +H100,FA3,1,524288,4,all_gather,54.211389801930636,swa1024,FA3 3.0.0 +H100,FA3,1,524288,8,a2a,23.195035121170804,swa1024,FA3 3.0.0 +H100,FA3,1,524288,8,all_gather,44.642224591225386,swa1024,FA3 3.0.0 +H100,FusedAttention,16,32768,2,a2a,80.653393808752298,swa1024,cuDNN 9.27.0 +H100,FusedAttention,16,32768,2,all_gather,470.50485660322011,swa1024,cuDNN 9.27.0 +H100,FusedAttention,16,32768,4,a2a,44.270377298817039,swa1024,cuDNN 9.27.0 +H100,FusedAttention,16,32768,4,all_gather,230.95962470397353,swa1024,cuDNN 9.27.0 +H100,FusedAttention,16,32768,8,a2a,24.141559917479753,swa1024,cuDNN 9.27.0 +H100,FusedAttention,16,32768,8,all_gather,126.96166828274727,swa1024,cuDNN 9.27.0 +H100,FusedAttention,8,65536,2,a2a,80.906134657561779,swa1024,cuDNN 9.27.0 +H100,FusedAttention,8,65536,2,all_gather,882.76211222633719,swa1024,cuDNN 9.27.0 +H100,FusedAttention,8,65536,4,a2a,44.434630628675222,swa1024,cuDNN 9.27.0 +H100,FusedAttention,8,65536,4,all_gather,418.89289688318968,swa1024,cuDNN 9.27.0 +H100,FusedAttention,8,65536,8,a2a,24.192158933728933,swa1024,cuDNN 9.27.0 +H100,FusedAttention,8,65536,8,all_gather,214.96841678395867,swa1024,cuDNN 9.27.0 +H100,FusedAttention,4,131072,2,a2a,81.114544263109565,swa1024,cuDNN 9.27.0 +H100,FusedAttention,4,131072,2,all_gather,1719.8689662851393,swa1024,cuDNN 9.27.0 +H100,FusedAttention,4,131072,4,a2a,44.613843159750104,swa1024,cuDNN 9.27.0 +H100,FusedAttention,4,131072,4,all_gather,792.05614109523594,swa1024,cuDNN 9.27.0 +H100,FusedAttention,4,131072,8,a2a,24.227923359721899,swa1024,cuDNN 9.27.0 +H100,FusedAttention,4,131072,8,all_gather,392.2153801843524,swa1024,cuDNN 9.27.0 +H100,FusedAttention,2,262144,2,a2a,81.412815749645233,swa1024,cuDNN 9.27.0 +H100,FusedAttention,2,262144,2,all_gather,3366.7698988225311,swa1024,cuDNN 9.27.0 +H100,FusedAttention,2,262144,4,a2a,44.487187592312694,swa1024,cuDNN 9.27.0 +H100,FusedAttention,2,262144,4,all_gather,1546.707306439057,swa1024,cuDNN 9.27.0 +H100,FusedAttention,2,262144,8,a2a,24.243694758042693,swa1024,cuDNN 9.27.0 +H100,FusedAttention,2,262144,8,all_gather,745.22636419162154,swa1024,cuDNN 9.27.0 +H100,FusedAttention,1,524288,2,a2a,81.036016223952174,swa1024,cuDNN 9.27.0 +H100,FusedAttention,1,524288,2,all_gather,6680.2752913534641,swa1024,cuDNN 9.27.0 +H100,FusedAttention,1,524288,4,a2a,44.550531785935163,swa1024,cuDNN 9.27.0 +H100,FusedAttention,1,524288,4,all_gather,3043.8656545616686,swa1024,cuDNN 9.27.0 +H100,FusedAttention,1,524288,8,a2a,24.227671287953854,swa1024,cuDNN 9.27.0 +H100,FusedAttention,1,524288,8,all_gather,1453.1953134387732,swa1024,cuDNN 9.27.0 diff --git a/benchmarks/attention/results/context_parallel/images/cp_backend_latencies_b200_review_v1.webp b/benchmarks/attention/results/context_parallel/images/cp_backend_latencies_b200_review_v1.webp new file mode 100644 index 00000000000..0e1a37357b2 Binary files /dev/null and b/benchmarks/attention/results/context_parallel/images/cp_backend_latencies_b200_review_v1.webp differ diff --git a/benchmarks/attention/results/context_parallel/images/cp_backend_latencies_h100_review_v1.webp b/benchmarks/attention/results/context_parallel/images/cp_backend_latencies_h100_review_v1.webp new file mode 100644 index 00000000000..f09643e679f Binary files /dev/null and b/benchmarks/attention/results/context_parallel/images/cp_backend_latencies_h100_review_v1.webp differ diff --git a/benchmarks/attention/results/context_parallel/images/cp_causal_endpoint_sweep_by_backend_review_v2.webp b/benchmarks/attention/results/context_parallel/images/cp_causal_endpoint_sweep_by_backend_review_v2.webp new file mode 100644 index 00000000000..c39e9bf1379 Binary files /dev/null and b/benchmarks/attention/results/context_parallel/images/cp_causal_endpoint_sweep_by_backend_review_v2.webp differ diff --git a/benchmarks/attention/results/context_parallel/images/cp_communication_dotplot_b200_review_v4.webp b/benchmarks/attention/results/context_parallel/images/cp_communication_dotplot_b200_review_v4.webp new file mode 100644 index 00000000000..c3c83350b8b Binary files /dev/null and b/benchmarks/attention/results/context_parallel/images/cp_communication_dotplot_b200_review_v4.webp differ diff --git a/benchmarks/attention/results/context_parallel/images/cp_communication_dotplot_h100_review_v4.webp b/benchmarks/attention/results/context_parallel/images/cp_communication_dotplot_h100_review_v4.webp new file mode 100644 index 00000000000..d1b2e1df456 Binary files /dev/null and b/benchmarks/attention/results/context_parallel/images/cp_communication_dotplot_h100_review_v4.webp differ diff --git a/benchmarks/attention/results/context_parallel/images/cp_swa_comm_clear_labels_review_v3.webp b/benchmarks/attention/results/context_parallel/images/cp_swa_comm_clear_labels_review_v3.webp new file mode 100644 index 00000000000..b784383dc4a Binary files /dev/null and b/benchmarks/attention/results/context_parallel/images/cp_swa_comm_clear_labels_review_v3.webp differ diff --git a/tests/pytorch/attention/benchmark_cp.py b/tests/pytorch/attention/benchmark_cp.py new file mode 100644 index 00000000000..29fbcb5d86e --- /dev/null +++ b/tests/pytorch/attention/benchmark_cp.py @@ -0,0 +1,469 @@ +# Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# See LICENSE for license information. + +"""Benchmark/profile configurations and cross-backend consistency test for CP attention. + +Configs here are intended for benchmarking and stress testing on top of the +core correctness suite in test_attention_with_cp.py. Most configs use larger +batch sizes / sequence lengths than the core suite. Variable-length THD inputs +use the `thd_seqlen_pattern` attribute, plumbed through to run_attention_with_cp.py. + +The bench/profile machinery (timing loop, cudaProfilerStart/Stop, save dir, seqlen +pattern arg) lives in run_attention_with_cp.py. +""" + +import os +import sys +import pathlib +import logging +import copy +import subprocess +import tempfile +import pytest +import torch +from transformer_engine.pytorch import ( + get_device_compute_capability, + get_cudnn_version, +) + +_current_file = pathlib.Path(__file__).resolve() +sys.path.append(str(_current_file.parent.parent)) +from utils import ModelConfig, get_available_attention_backends, run_distributed + +pytest_logging_level = logging.getLevelName(logging.root.level) + + +def get_bash_arguments(num_gpus_per_node, **kwargs): + args = [ + sys.executable, + "-m", + "torch.distributed.run", + "--standalone", + "--nproc-per-node=" + str(num_gpus_per_node), + ] + te_path = os.getenv("TE_PATH", "/opt/transformerengine") + args.append(os.path.join(te_path, "tests/pytorch/attention/run_attention_with_cp.py")) + for k, v in kwargs.items(): + args.append(f"{k}={v}") + return args + + +# Benchmark/stress configs (llama3_8b-like: 32 heads, 8 GQA, d=128). +model_configs_fused_attn = { + # Llama3-8b-shaped, varying seqlen + "bench_8k": ModelConfig(2, 8192, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "bench_16k": ModelConfig(1, 16384, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "bench_32k": ModelConfig(1, 32768, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + # Fixed 512K tokens with different sequence-count/length decompositions. + "uniform_1x512k": ModelConfig(1, 524288, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_2x256k": ModelConfig(2, 262144, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_4x128k": ModelConfig(4, 131072, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_8x64k": ModelConfig(8, 65536, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_16x32k": ModelConfig(16, 32768, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_32x16k": ModelConfig(32, 16384, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_64x8k": ModelConfig(64, 8192, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_128x4k": ModelConfig(128, 4096, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "uniform_256x2k": ModelConfig(256, 2048, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + # Fixed-token SWA probes use the same shapes with a 1024-token left window. + "uniform_1x512k_swa1024": ModelConfig( + 1, + 524288, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + "uniform_2x256k_swa1024": ModelConfig( + 2, + 262144, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + "uniform_4x128k_swa1024": ModelConfig( + 4, + 131072, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + "uniform_8x64k_swa1024": ModelConfig( + 8, + 65536, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + "uniform_16x32k_swa1024": ModelConfig( + 16, + 32768, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + "uniform_1x128k": ModelConfig( + 1, + 131072, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + ), + "uniform_1x128k_swa1024": ModelConfig( + 1, + 131072, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + # 2688-wide model with 32 query heads: 2688 / 32 = 84 channels per head. + "uniform_1x128k_h32_d2688": ModelConfig( + 1, + 131072, + 32, + 84, + num_gqa_groups=8, + attn_mask_type="causal", + ), + "uniform_1x128k_h32_d2688_swa1024": ModelConfig( + 1, + 131072, + 32, + 84, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + # THD stress: higher batch / longer seqlen than core suite + "cp_thd_0": ModelConfig(8, 8192, 12, 128, attn_mask_type="causal"), # MHA b=8 + "cp_thd_1": ModelConfig(8, 8192, 12, 128), # MHA b=8 non-causal + "cp_thd_2": ModelConfig(16, 4096, 12, 128, attn_mask_type="causal"), # MHA b=16 + "cp_thd_3": ModelConfig(8, 8192, 12, 128, num_gqa_groups=2, attn_mask_type="causal"), # GQA b=8 + # THD + SWA + "cp_thd_swa_0": ModelConfig( + 8, 8192, 12, 128, attn_mask_type="causal", window_size=(512, 0) + ), # MHA SWA causal + "cp_thd_swa_1": ModelConfig( + 8, 8192, 12, 128, num_gqa_groups=2, attn_mask_type="causal", window_size=(512, 0) + ), # GQA SWA causal + "cp_thd_swa_2": ModelConfig( + 8, 8192, 12, 128, attn_mask_type="causal", window_size=(512, 512) + ), # MHA SWA causal+right + "cp_thd_swa_3": ModelConfig( + 8, 8192, 12, 128, num_gqa_groups=2, attn_mask_type="causal", window_size=(512, 512) + ), # GQA SWA causal+right +} + + +# Variable-length training-workload configs used to exercise non-uniform THD batches. +_training_workloads = { + "bucket32k": ( + ModelConfig(4, 32768, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "24576,28672,30720,32768", + ), + "bucket64k": ( + ModelConfig(4, 65536, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "57344,61440,63488,65536", + ), + "mixed32k": ( + ModelConfig(8, 32768, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "16384,24576,32768,8192,28672,32768,20480,16384", + ), + "rl16k": ( + ModelConfig(8, 16384, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "4096,6144,6144,8192,8192,10240,12288,16384", + ), + "outlier64k": ( + ModelConfig(4, 65536, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "8192,8192,8192,65536", + ), + "bucket128k": ( + ModelConfig(3, 131072, 32, 128, num_gqa_groups=8, attn_mask_type="causal"), + "114688,122880,131072", + ), + # SWA variants of mixed32k for cross-backend SWA correctness checks. + "mixed32k_swa512": ( + ModelConfig( + 8, + 32768, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(512, 0), + ), + "16384,24576,32768,8192,28672,32768,20480,16384", + ), + "mixed32k_swa1024": ( + ModelConfig( + 8, + 32768, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(1024, 0), + ), + "16384,24576,32768,8192,28672,32768,20480,16384", + ), + "mixed32k_swa2048": ( + ModelConfig( + 8, + 32768, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(2048, 0), + ), + "16384,24576,32768,8192,28672,32768,20480,16384", + ), +} + +# SWA variants of all 6 training workloads at windows 512/1024/2048. +_swa_base = { + "bucket32k": (4, 32768, "24576,28672,30720,32768"), + "bucket64k": (4, 65536, "57344,61440,63488,65536"), + "mixed32k_full": (8, 32768, "16384,24576,32768,8192,28672,32768,20480,16384"), + "rl16k": (8, 16384, "4096,6144,6144,8192,8192,10240,12288,16384"), + "outlier64k": (4, 65536, "8192,8192,8192,65536"), + "bucket128k": (3, 131072, "114688,122880,131072"), +} +for _name, (_b, _s, _pat) in _swa_base.items(): + for _w in (512, 1024, 2048): + # Use "_full" suffix dropped; keep mixed32k_swa* names matching above + if _name == "mixed32k_full": + continue # already added with mixed32k_swa{512,1024,2048} + _key = f"{_name}_swa{_w}" + _training_workloads[_key] = ( + ModelConfig( + _b, + _s, + 32, + 128, + num_gqa_groups=8, + attn_mask_type="causal", + window_size=(_w, 0), + ), + _pat, + ) +for _name, (_cfg, _pat) in _training_workloads.items(): + _cfg.thd_seqlen_pattern = _pat + model_configs_fused_attn[_name] = _cfg + + +# Pytest-runnable subset: skip the largest fixed-token SWA configs. +_pytest_skip_configs = { + "uniform_1x512k_swa1024", + "uniform_2x256k_swa1024", + "uniform_16x32k_swa1024", +} +_pytest_configs = { + k: v for k, v in model_configs_fused_attn.items() if k not in _pytest_skip_configs +} + + +@pytest.mark.skipif(get_cudnn_version() < (8, 9, 7), reason="cuDNN 8.9.7+ is required.") +@pytest.mark.skipif(get_device_compute_capability() < (9, 0), reason="THD requires sm90+.") +@pytest.mark.parametrize("model", _pytest_configs.keys()) +@pytest.mark.parametrize("qkv_format", ["thd"]) +@pytest.mark.parametrize("cp_comm_type", ["p2p", "all_gather", "a2a"]) +def test_cp_benchmark_configs(model, qkv_format, cp_comm_type): + """Run benchmark/stress configs through the core CP path for correctness.""" + if 2 > torch.cuda.device_count(): + pytest.skip("Test requires 2 GPUs") + + config = _pytest_configs[model] + config.context_parallel = True + config.cp_comm_type = cp_comm_type + + has_swa = config.window_size != (-1, 0) and config.window_size != (-1, -1) + if has_swa and cp_comm_type == "p2p": + pytest.skip("p2p does not support sliding window") + + # THD uses padding mask types for backend availability check + check_config = copy.deepcopy(config) + if "causal" in check_config.attn_mask_type: + check_config.attn_mask_type = "padding_causal" + else: + check_config.attn_mask_type = "padding" + + available, *_ = get_available_attention_backends( + check_config, + qkv_dtype=torch.bfloat16, + qkv_layout="_".join([qkv_format] * 3), + ) + _, fused_supported, _ = available + if not fused_supported: + pytest.skip("FusedAttention not available for this config") + + extra_kwargs = {} + thd_pat = getattr(config, "thd_seqlen_pattern", None) + if thd_pat is not None: + extra_kwargs["thd_seqlen_pattern"] = thd_pat + + run_distributed( + get_bash_arguments( + num_gpus_per_node=2, + dtype="bf16", + model=model, + qkv_format=qkv_format, + kernel_backend="FusedAttention", + cp_comm_type=cp_comm_type, + log_level=pytest_logging_level, + **extra_kwargs, + ), + ) + + +@pytest.mark.parametrize("qkv_format", ["bshd", "sbhd"]) +@pytest.mark.parametrize("model,is_training", [("cp_1_2", True), ("cp_2_4", False)]) +def test_cp_benchmark_only_bias(model, is_training, qkv_format): + """Cover partitioned and broadcast bias without allocating full Q/K/V inputs.""" + from test_attention_with_cp import model_configs_fused_attn as core_configs + + if torch.cuda.device_count() < 2: + pytest.skip("Test requires 2 GPUs") + config = copy.deepcopy(core_configs[model]) + config.context_parallel = True + config.cp_comm_type = "p2p" + available, *_ = get_available_attention_backends( + config, + qkv_dtype=torch.bfloat16, + qkv_layout="_".join([qkv_format] * 3), + is_training=is_training, + cp_size=2, + ) + if not available[1]: + pytest.skip("FusedAttention not available for this bias configuration") + + env = os.environ.copy() + env["NVTE_CP_BENCH_ONLY"] = "1" + result = run_distributed( + get_bash_arguments( + num_gpus_per_node=2, + dtype="bf16", + model=model, + qkv_format=qkv_format, + kernel_backend="FusedAttention", + cp_comm_type="p2p", + is_training=is_training, + benchmark=1, + log_level=pytest_logging_level, + ), + env=env, + stdout=subprocess.PIPE, + ) + assert "CP_BENCH_ONLY correctness_paths=skipped inputs=rank_local" in result.stdout + assert "CP_BENCH_RESULT " in result.stdout + + +# Cross-backend consistency configs: run the same input through p2p / all_gather / a2a +# and assert the outputs agree within tolerance. +model_configs_cross_backend = { + "cp_thd_0": model_configs_fused_attn["cp_thd_0"], + "cp_thd_1": model_configs_fused_attn["cp_thd_1"], + "cp_thd_2": model_configs_fused_attn["cp_thd_2"], + "cp_thd_3": model_configs_fused_attn["cp_thd_3"], + "cp_thd_swa_0": model_configs_fused_attn["cp_thd_swa_0"], + "cp_thd_swa_1": model_configs_fused_attn["cp_thd_swa_1"], + "cp_thd_swa_2": model_configs_fused_attn["cp_thd_swa_2"], + "cp_thd_swa_3": model_configs_fused_attn["cp_thd_swa_3"], +} +# Add a few training workloads (smaller ones to keep runtime reasonable) +for _name in ["rl16k", "bucket32k", "mixed32k"]: + if _name in model_configs_fused_attn: + model_configs_cross_backend[_name] = model_configs_fused_attn[_name] + + +@pytest.mark.skipif(get_cudnn_version() < (9, 3, 0), reason="cuDNN 9.3.0+ is required.") +@pytest.mark.skipif(get_device_compute_capability() < (9, 0), reason="THD requires sm90+.") +@pytest.mark.parametrize("model", model_configs_cross_backend.keys()) +def test_cp_thd_cross_backend_consistency(model): + """Compare outputs of p2p, all_gather, and a2a backends for THD format.""" + if 2 > torch.cuda.device_count(): + pytest.skip("Test requires 2 GPUs") + + config = model_configs_cross_backend[model] + config.context_parallel = True + + has_swa = config.window_size != (-1, 0) and config.window_size != (-1, -1) + # p2p doesn't support sliding window + backends = ["all_gather", "a2a"] if has_swa else ["p2p", "all_gather", "a2a"] + saved_outputs = {} + + # THD uses padding mask types for backend availability check + check_config = copy.deepcopy(config) + if "causal" in check_config.attn_mask_type: + check_config.attn_mask_type = "padding_causal" + else: + check_config.attn_mask_type = "padding" + + with tempfile.TemporaryDirectory() as tmpdir: + for backend in backends: + check_config.cp_comm_type = backend + available, *_ = get_available_attention_backends( + check_config, + qkv_dtype=torch.bfloat16, + qkv_layout="thd_thd_thd", + ) + _, fused_supported, _ = available + if not fused_supported: + pytest.skip(f"FusedAttention not available for {backend}") + + save_dir = os.path.join(tmpdir, backend) + env = os.environ.copy() + env["CP_CROSS_BACKEND_SAVE_DIR"] = save_dir + extra_kwargs = {} + thd_pat = getattr(config, "thd_seqlen_pattern", None) + if thd_pat is not None: + extra_kwargs["thd_seqlen_pattern"] = thd_pat + run_distributed( + get_bash_arguments( + num_gpus_per_node=2, + dtype="bf16", + model=model, + qkv_format="thd", + kernel_backend="FusedAttention", + cp_comm_type=backend, + log_level=pytest_logging_level, + **extra_kwargs, + ), + env=env, + ) + saved_outputs[backend] = { + r: torch.load( + os.path.join(save_dir, f"outputs_{backend}_rank{r}.pt"), + weights_only=True, + ) + for r in range(2) + } + + # Compare all backends pairwise against the first as reference. + # Cross-backend diffs compound two independent CP implementations, + # so use a wider tolerance than the per-backend CP-vs-nonCP tests. + ref = backends[0] + atol = 0.1 + for backend in backends[1:]: + for rank in range(2): + ref_out = saved_outputs[ref][rank] + cmp_out = saved_outputs[backend][rank] + for key in ["out", "dq", "dk", "dv"]: + if ref_out[key] is None or cmp_out[key] is None: + continue + diff = (ref_out[key] - cmp_out[key]).abs().max().item() + assert ( + diff < atol + ), f"{backend} vs {ref} rank{rank} {key}: max_diff={diff} > {atol}" diff --git a/tests/pytorch/attention/run_attention_with_cp.py b/tests/pytorch/attention/run_attention_with_cp.py index ef4b9499ebe..8c81706c8e3 100644 --- a/tests/pytorch/attention/run_attention_with_cp.py +++ b/tests/pytorch/attention/run_attention_with_cp.py @@ -3,8 +3,10 @@ # See LICENSE for license information. import copy +import json import os import sys +import time import logging from contextlib import nullcontext import torch @@ -38,6 +40,16 @@ ) from utils import ModelConfig, compare_and_assert +# Keep benchmark-only workload aliases out of the core correctness suite. +try: + from benchmark_cp import model_configs_fused_attn as _bench_cfgs_fused_attn + + for _key, _config in _bench_cfgs_fused_attn.items(): + model_configs_fused_attn.setdefault(_key, _config) + model_configs_flash_attn.setdefault(_key, _config) +except ImportError: + pass + # Pool mode (NVTE_CP_POOL_PG=1) only: shared CP collective groups, created once # per pool by run_attention_with_cp_pool.main() and reused across every case in # that pool. world_size and the rank set don't change per case, so re-creating @@ -57,6 +69,7 @@ def generate_input_shapes( kernel_backend: str, fa_pad_between_seqs: str = "False", load_balancing_strategy=CPLoadBalancingStrategy.DUAL_CHUNK_SWAP, + thd_seqlen_pattern: str = "random", ): if qkv_format == "bshd": q_input_shape = ( @@ -136,9 +149,38 @@ def generate_input_shapes( dtype=torch.int32, ) else: - seqlens_q = torch.randint(0, config.max_seqlen_q + 1, [config.batch_size]).to( - torch.int32 - ) + batch_size = config.batch_size + max_seqlen = config.max_seqlen_q + if "," in thd_seqlen_pattern: + explicit_seqlens = [int(length) for length in thd_seqlen_pattern.split(",")] + if not explicit_seqlens or any(length <= 0 for length in explicit_seqlens): + raise ValueError("Explicit THD sequence lengths must all be positive") + if max(explicit_seqlens) > max_seqlen: + raise ValueError( + "Explicit THD sequence lengths must not exceed the configured maximum" + ) + seqlens_q = torch.tensor( + explicit_seqlens, + dtype=torch.int32, + ) + config.batch_size = len(seqlens_q) + config.max_seqlen_q = int(seqlens_q.max()) + config.max_seqlen_kv = config.max_seqlen_q + elif thd_seqlen_pattern == "max": + seqlens_q = torch.full([batch_size], max_seqlen, dtype=torch.int32) + elif thd_seqlen_pattern == "half": + seqlens_q = torch.full([batch_size], max_seqlen // 2, dtype=torch.int32) + elif thd_seqlen_pattern == "linear": + seqlens_q = torch.linspace(1, max_seqlen, batch_size).to(torch.int32) + elif thd_seqlen_pattern == "alternating": + seqlens_q = torch.tensor( + [max_seqlen if i % 2 == 0 else max_seqlen // 4 for i in range(batch_size)], + dtype=torch.int32, + ) + elif thd_seqlen_pattern == "random": + seqlens_q = torch.randint(0, max_seqlen + 1, [batch_size], dtype=torch.int32) + else: + raise ValueError(f"Unsupported THD sequence-length pattern: {thd_seqlen_pattern}") seqlens_q_padded = ( (seqlens_q + 2 * world_size - 1) // (world_size * 2) * (world_size * 2) ) @@ -323,10 +365,19 @@ def run_dpa_with_cp( deterministic="False", load_balancing_strategy="DUAL_CHUNK_SWAP", softcap="0.0", + benchmark="0", + thd_seqlen_pattern="random", log_level=logging.WARNING, ): """Test DotProductAttention module with context parallelism""" + torch.manual_seed(1234) logging.root.setLevel(log_level) + benchmark_iters = int(benchmark) + cp_bench_only = os.getenv("NVTE_CP_BENCH_ONLY", "0") == "1" + if cp_bench_only and benchmark_iters <= 0: + raise ValueError("NVTE_CP_BENCH_ONLY requires benchmark > 0") + if cp_bench_only and int(os.getenv("RANK", "0")) == 0: + print("CP_BENCH_ONLY correctness_paths=skipped inputs=rank_local", flush=True) load_balancing_strategy = CPLoadBalancingStrategy[load_balancing_strategy] # When is_training is False, gradient outputs are None. is_training = is_training == "True" @@ -345,6 +396,8 @@ def run_dpa_with_cp( os.environ["NVTE_FP8_DPA_BWD"] = "1" if fp8_bwd else "0" fp8_dpa = fp8_dpa == "True" and dtype == "fp8" fp8_mha = fp8_mha == "True" and dtype == "fp8" and scaling_mode != "mxfp8" + if benchmark_iters > 0 and dtype == "fp8": + raise ValueError("CP benchmark mode currently supports fp16 and bf16 only") f16_O = dtype == "fp8" and scaling_mode in ["current", "mxfp8"] and f16_O == "True" os.environ["NVTE_DPA_FP8CS_O_in_F16"] = "1" if f16_O else "0" os.environ["NVTE_FLASH_ATTN"] = "0" @@ -385,6 +438,8 @@ def run_dpa_with_cp( device_count = torch.cuda.device_count() device = rank % device_count torch.cuda.set_device(device) + # Seed after selecting the rank-local device so every process seeds the GPU it uses. + torch.cuda.manual_seed(1234) logging.info(f"[Rank {rank}] Setup: world_size {world_size}") if not _pool_managed_pg: dist.init_process_group(backend="nccl", world_size=world_size, rank=rank) @@ -458,13 +513,65 @@ def run_dpa_with_cp( kernel_backend, fa_pad_between_seqs, load_balancing_strategy, + thd_seqlen_pattern, ) - q_orig = torch.clamp(torch.randn(q_input_shape, dtype=dtypes[dtype]), min=-1, max=1).cuda() - k_orig = torch.clamp(torch.randn(k_input_shape, dtype=dtypes[dtype]), min=-1, max=1).cuda() - v_orig = torch.clamp(torch.randn(v_input_shape, dtype=dtypes[dtype]), min=-1, max=1).cuda() - dout_orig = torch.clamp( - torch.randn(attn_output_shape, dtype=dtypes[dtype]), min=-1, max=1 - ).cuda() + input_shapes = [q_input_shape, k_input_shape, v_input_shape, attn_output_shape] + if qkv_format in ("bshd", "sbhd"): + # Dense inputs and bias use the same two chunks in both execution modes. + seq_idx = torch.tensor([rank, 2 * world_size - rank - 1], device="cuda") + if cp_bench_only: + if qkv_format in ("bshd", "sbhd"): + seq_dim = qkv_format.index("s") + input_shapes = [list(shape) for shape in input_shapes] + for shape in input_shapes: + shape[seq_dim] //= world_size + elif qkv_format == "thd": + seq_idx_q = get_thd_partitioned_indices( + cu_seqlens_q_padded, + int(q_input_shape[0]), + world_size, + rank, + device="cuda", + load_balancing_strategy=load_balancing_strategy, + ) + seq_idx_kv = get_thd_partitioned_indices( + cu_seqlens_kv_padded, + int(k_input_shape[0]), + world_size, + rank, + device="cuda", + load_balancing_strategy=load_balancing_strategy, + ) + input_shapes = [ + (seq_idx_q.numel(), *q_input_shape[1:]), + (seq_idx_kv.numel(), *k_input_shape[1:]), + (seq_idx_kv.numel(), *v_input_shape[1:]), + (seq_idx_q.numel(), *attn_output_shape[1:]), + ] + q_, k_, v_, dout_ = [ + torch.clamp(torch.randn(shape, dtype=dtypes[dtype]), min=-1, max=1).cuda() + for shape in input_shapes + ] + else: + q_orig, k_orig, v_orig, dout_orig = [ + torch.clamp(torch.randn(shape, dtype=dtypes[dtype]), min=-1, max=1).cuda() + for shape in input_shapes + ] + + _save_path = os.environ.get("CP_CROSS_BACKEND_SAVE_DIR") + if _save_path and not cp_bench_only: + os.makedirs(_save_path, exist_ok=True) + torch.save( + { + "q": q_orig, + "k": k_orig, + "v": v_orig, + "dout": dout_orig, + "cu_seqlens_q": cu_seqlens_q, + "cu_seqlens_q_padded": cu_seqlens_q_padded, + }, + os.path.join(_save_path, f"inputs_rank{rank}.pt"), + ) if scaling_mode == "delayed": qkv_quantizer = Float8Quantizer( fp8_dtype=DType.kFloat8E4M3, @@ -501,11 +608,12 @@ def run_dpa_with_cp( dout_quantizer.optimize_for_gemm = True dout_quantizer.internal = False qkv_layout = "_".join([qkv_format] * 3) - q, k, v, dout = [x.clone().detach() for x in [q_orig, k_orig, v_orig, dout_orig]] - if fp8_mha: - q, k, v, qkv_layout, _ = combine_and_quantize(qkv_layout, q, k, v, qkv_quantizer) - for x in [q, k, v]: - x.requires_grad = True + if not cp_bench_only: + q, k, v, dout = [x.clone().detach() for x in [q_orig, k_orig, v_orig, dout_orig]] + if fp8_mha: + q, k, v, qkv_layout, _ = combine_and_quantize(qkv_layout, q, k, v, qkv_quantizer) + for x in [q, k, v]: + x.requires_grad = True if config.attn_bias_type not in ["no_bias", "alibi"]: bias_shape_map = { @@ -530,97 +638,101 @@ def run_dpa_with_cp( else: bias = None - ############ run without CP ############ - logging.info(f"[Rank {rank}] Run without context parallelism") if dtype == "fp8": fp8_context = autocast(enabled=True, recipe=fp8_recipe, amax_reduction_group=cp_comm_group) else: fp8_context = nullcontext() - max_logit = None - with fp8_context: - # q, k, v, out in FP8; dout in F16 - out = core_attn( - q, - k, - v, - core_attention_bias_type=config.attn_bias_type, - core_attention_bias=bias, - cu_seqlens_q=cu_seqlens_q, - cu_seqlens_kv=cu_seqlens_kv, - cu_seqlens_q_padded=cu_seqlens_q_padded, - cu_seqlens_kv_padded=cu_seqlens_kv_padded, - pad_between_seqs=pad_between_seqs, - fp8_output=fp8_mha, - ) - if config.return_max_logit: - out, max_logit = out - # Anchor the non-CP result to something other than another run of the same backend, before - # it is reshaped for the CP comparison. See the function docstring for why this exists. - assert_matches_independent_reference( - out, q, k, v, config, qkv_format, dtype, core_attn.softmax_scale - ) + if not cp_bench_only: + ############ run without CP ############ + logging.info(f"[Rank {rank}] Run without context parallelism") + max_logit = None + with fp8_context: + # q, k, v, out in FP8; dout in F16 + out = core_attn( + q, + k, + v, + core_attention_bias_type=config.attn_bias_type, + core_attention_bias=bias, + cu_seqlens_q=cu_seqlens_q, + cu_seqlens_kv=cu_seqlens_kv, + cu_seqlens_q_padded=cu_seqlens_q_padded, + cu_seqlens_kv_padded=cu_seqlens_kv_padded, + pad_between_seqs=pad_between_seqs, + fp8_output=fp8_mha, + ) + if config.return_max_logit: + out, max_logit = out + # Preserve the independent reference for correctness runs, while benchmark-only + # execution continues to skip the full non-CP reference path. + assert_matches_independent_reference( + out, q, k, v, config, qkv_format, dtype, core_attn.softmax_scale + ) + if is_training: + if fp8_bwd and fp8_mha: + dout_fp8 = dout_quantizer(dout) + out.backward(dout_fp8) + else: + out.backward(dout) if is_training: - if fp8_bwd and fp8_mha: - dout_fp8 = dout_quantizer(dout) - out.backward(dout_fp8) - else: - out.backward(dout) - if is_training: - dq, dk, dv, dbias = q.grad, k.grad, v.grad, bias.grad if bias is not None else None - d_softmax_offset = ( - core_attn.softmax_offset.grad if config.softmax_type != "vanilla" else None - ) - else: - dq, dk, dv, dbias = None, None, None, None - d_softmax_offset = None + dq, dk, dv, dbias = q.grad, k.grad, v.grad, bias.grad if bias is not None else None + d_softmax_offset = ( + core_attn.softmax_offset.grad if config.softmax_type != "vanilla" else None + ) + else: + dq, dk, dv, dbias = None, None, None, None + d_softmax_offset = None ############ run with CP ############ logging.info(f"[Rank {rank}] Run with context parallelism") # set up inputs - q_, k_, v_, dout_, *rest = [ - x.clone().detach() - for x in [q_orig, k_orig, v_orig, dout_orig] + ([] if bias is None else [bias]) - ] - bias_ = rest[0] if len(rest) else None - if qkv_format in ("bshd", "sbhd"): - seq_dim = qkv_format.index("s") - q_, k_, v_, dout_ = [ - x.view( - *x.shape[:seq_dim], - 2 * world_size, - x.shape[seq_dim] // (2 * world_size), - *x.shape[(seq_dim + 1) :], + bias_ = bias.clone().detach() if bias is not None else None + if not cp_bench_only: + q_, k_, v_, dout_ = [x.clone().detach() for x in [q_orig, k_orig, v_orig, dout_orig]] + if qkv_format in ("bshd", "sbhd"): + seq_dim = qkv_format.index("s") + q_, k_, v_, dout_ = [ + x.view( + *x.shape[:seq_dim], + 2 * world_size, + x.shape[seq_dim] // (2 * world_size), + *x.shape[(seq_dim + 1) :], + ) + for x in [q_, k_, v_, dout_] + ] + q_, k_, v_, dout_ = [x.index_select(seq_dim, seq_idx) for x in [q_, k_, v_, dout_]] + q_, k_, v_, dout_ = [ + x.view(*x.shape[:seq_dim], -1, *x.shape[(seq_dim + 2) :]) + for x in [q_, k_, v_, dout_] + ] + elif qkv_format == "thd": + seq_idx_q = get_thd_partitioned_indices( + cu_seqlens_q_padded, + q_.shape[0], + world_size, + rank, + device=q_.device, + load_balancing_strategy=load_balancing_strategy, ) - for x in [q_, k_, v_, dout_] - ] - seq_idx = torch.tensor([rank, 2 * world_size - rank - 1], device=q_.device) - q_, k_, v_, dout_ = [x.index_select(seq_dim, seq_idx) for x in [q_, k_, v_, dout_]] - q_, k_, v_, dout_ = [ - x.view(*x.shape[:seq_dim], -1, *x.shape[(seq_dim + 2) :]) for x in [q_, k_, v_, dout_] - ] - elif qkv_format == "thd": - seq_idx_q = get_thd_partitioned_indices( - cu_seqlens_q_padded, - q_.shape[0], - world_size, - rank, - device=q_.device, - load_balancing_strategy=load_balancing_strategy, - ) - seq_idx_kv = get_thd_partitioned_indices( - cu_seqlens_kv_padded, - k_.shape[0], - world_size, - rank, - device=k_.device, - load_balancing_strategy=load_balancing_strategy, - ) - q_, dout_ = [x.index_select(0, seq_idx_q) for x in [q_, dout_]] - k_, v_ = [x.index_select(0, seq_idx_kv) for x in [k_, v_]] - else: - assert False, f"{qkv_format} is an unsupported qkv_format!" + seq_idx_kv = get_thd_partitioned_indices( + cu_seqlens_kv_padded, + k_.shape[0], + world_size, + rank, + device=k_.device, + load_balancing_strategy=load_balancing_strategy, + ) + q_, dout_ = [x.index_select(0, seq_idx_q) for x in [q_, dout_]] + k_, v_ = [x.index_select(0, seq_idx_kv) for x in [k_, v_]] + else: + assert False, f"{qkv_format} is an unsupported qkv_format!" q_, k_, v_, dout_ = [x.contiguous() for x in [q_, k_, v_, dout_]] + if not cp_bench_only: + out = out.detach() + if max_logit is not None: + max_logit = max_logit.detach() + del q, k, v, dout, q_orig, k_orig, v_orig, dout_orig if scaling_mode == "delayed": qkv_quantizer.scale.fill_(1.0) qkv_quantizer.amax.fill_(0.0) @@ -633,10 +745,6 @@ def run_dpa_with_cp( if bias_ is not None: ndim = bias_.ndim seq_q_dim = ndim - 2 - if qkv_format == "thd": - bias_seq_idx = seq_idx_q - else: - bias_seq_idx = seq_idx shape_before_seq = bias_.shape[:seq_q_dim] seq_q_size = bias_.shape[seq_q_dim] seq_kv_size = bias_.shape[-1] @@ -646,6 +754,7 @@ def run_dpa_with_cp( # Bias is broadcast, no need to partition along sequence dimension pass else: + bias_seq_idx = seq_idx_q if qkv_format == "thd" else seq_idx bias_ = bias_.view( *shape_before_seq, 2 * world_size, seq_q_size // (2 * world_size), seq_kv_size ) @@ -660,7 +769,11 @@ def run_dpa_with_cp( cp_comm_type, load_balancing_strategy, ) - if is_training and config.softmax_type != "vanilla": + if ( + is_training + and config.softmax_type != "vanilla" + and core_attn.softmax_offset.grad is not None + ): core_attn.softmax_offset.grad.zero_() if dtype == "fp8": core_attn.fp8_initialized = False @@ -669,44 +782,131 @@ def run_dpa_with_cp( else: fp8_context = nullcontext() - # run attention - max_logit_ = None - with fp8_context: - # q, k, v, out in FP8; dout in F16 - out_ = core_attn( - q_, - k_, - v_, - core_attention_bias_type=config.attn_bias_type, - core_attention_bias=bias_, - cu_seqlens_q=cu_seqlens_q, - cu_seqlens_kv=cu_seqlens_kv, - cu_seqlens_q_padded=cu_seqlens_q_padded, - cu_seqlens_kv_padded=cu_seqlens_kv_padded, - pad_between_seqs=pad_between_seqs, - fp8_output=fp8_mha, - ) - if config.return_max_logit: - out_, max_logit_ = out_ + if not cp_bench_only: + # This pass is for correctness; benchmark-only mode enters warmup directly. + max_logit_ = None + with fp8_context: + # q, k, v, out in FP8; dout in F16 + out_ = core_attn( + q_, + k_, + v_, + core_attention_bias_type=config.attn_bias_type, + core_attention_bias=bias_, + cu_seqlens_q=cu_seqlens_q, + cu_seqlens_kv=cu_seqlens_kv, + cu_seqlens_q_padded=cu_seqlens_q_padded, + cu_seqlens_kv_padded=cu_seqlens_kv_padded, + pad_between_seqs=pad_between_seqs, + fp8_output=fp8_mha, + ) + if config.return_max_logit: + out_, max_logit_ = out_ + if is_training: + if fp8_bwd and fp8_mha: + dout_fp8_ = dout_quantizer(dout_) + out_.backward(dout_fp8_) + else: + out_.backward(dout_) if is_training: - if fp8_bwd and fp8_mha: - dout_fp8_ = dout_quantizer(dout_) - out_.backward(dout_fp8_) - else: - out_.backward(dout_) - if is_training: - dq_, dk_, dv_, dbias_ = ( - q_.grad, - k_.grad, - v_.grad, - bias_.grad if bias_ is not None else None, - ) - d_softmax_offset_ = ( - core_attn.softmax_offset.grad.clone() if config.softmax_type != "vanilla" else None + dq_, dk_, dv_, dbias_ = ( + q_.grad, + k_.grad, + v_.grad, + bias_.grad if bias_ is not None else None, + ) + d_softmax_offset_ = ( + core_attn.softmax_offset.grad.clone() if config.softmax_type != "vanilla" else None + ) + else: + dq_, dk_, dv_, dbias_ = None, None, None, None + d_softmax_offset_ = None + + if _save_path and not cp_bench_only: + torch.save( + { + "out": out_.detach(), + "dq": dq_.detach() if dq_ is not None else None, + "dk": dk_.detach() if dk_ is not None else None, + "dv": dv_.detach() if dv_ is not None else None, + }, + os.path.join(_save_path, f"outputs_{cp_comm_type}_rank{rank}.pt"), ) - else: - dq_, dk_, dv_, dbias_ = None, None, None, None - d_softmax_offset_ = None + + # Input leaves are recreated outside the timed window. Rank-local durations + # are reduced after profiling so the endpoint metric is the mean of the + # per-iteration rank maxima, which represents distributed completion time. + if benchmark_iters > 0: + warmup = 10 + local_latencies_ms = [] + for iteration in range(warmup + benchmark_iters): + q_b, k_b, v_b = [x.clone().detach().requires_grad_() for x in (q_, k_, v_)] + if bias_ is not None: + bias_.grad = None + if config.softmax_type != "vanilla": + core_attn.softmax_offset.grad = None + dist.barrier(group=cp_comm_group) + torch.cuda.synchronize() + if iteration == warmup: + torch.cuda.cudart().cudaProfilerStart() + start = time.perf_counter() + with fp8_context: + out_b = core_attn( + q_b, + k_b, + v_b, + core_attention_bias_type=config.attn_bias_type, + core_attention_bias=bias_, + cu_seqlens_q=cu_seqlens_q, + cu_seqlens_kv=cu_seqlens_kv, + cu_seqlens_q_padded=cu_seqlens_q_padded, + cu_seqlens_kv_padded=cu_seqlens_kv_padded, + pad_between_seqs=pad_between_seqs, + fp8_output=fp8_mha, + ) + if isinstance(out_b, tuple): + out_b = out_b[0] + if is_training: + if fp8_bwd and fp8_mha: + out_b.backward(dout_quantizer(dout_)) + else: + out_b.backward(dout_) + torch.cuda.synchronize() + if iteration >= warmup: + local_latencies_ms.append((time.perf_counter() - start) * 1000) + del out_b, q_b, k_b, v_b + torch.cuda.cudart().cudaProfilerStop() + local_samples = torch.tensor(local_latencies_ms, dtype=torch.float64, device="cuda") + gathered_samples = [torch.empty_like(local_samples) for _ in range(world_size)] + dist.all_gather(gathered_samples, local_samples, group=cp_comm_group) + if rank == 0: + iteration_rank_max_ms = torch.stack(gathered_samples).max(dim=0).values.cpu().tolist() + result = { + "model": model, + "backend": kernel_backend, + "communication_type": cp_comm_type, + "qkv_format": qkv_format, + "dtype": dtype, + "cp_size": world_size, + "is_training": is_training, + "thd_seqlen_pattern": thd_seqlen_pattern, + "warmup_iterations": warmup, + "timed_iterations": benchmark_iters, + "statistic": "mean_of_iteration_rank_max_ms", + "latency_ms": sum(iteration_rank_max_ms) / benchmark_iters, + "iteration_rank_max_ms": iteration_rank_max_ms, + } + print(f"CP_BENCH_RESULT {json.dumps(result, sort_keys=True)}", flush=True) + + if cp_bench_only: + if not _reusing_pool_groups: + if cp_comm_group is not None: + dist.destroy_process_group(cp_comm_group) + for group in cp_comm_sub_groups: + dist.destroy_process_group(group) + if not _pool_managed_pg: + dist.destroy_process_group() + return # get outputs tensors = [out, dq, dk, dv, dbias, out_, dq_, dk_, dv_, dbias_] diff --git a/transformer_engine/pytorch/attention/dot_product_attention/context_parallel.py b/transformer_engine/pytorch/attention/dot_product_attention/context_parallel.py index 42fea51cecf..1545e933f05 100644 --- a/transformer_engine/pytorch/attention/dot_product_attention/context_parallel.py +++ b/transformer_engine/pytorch/attention/dot_product_attention/context_parallel.py @@ -11,6 +11,7 @@ from transformer_engine.pytorch.device import te_device_type from transformer_engine.pytorch.utils import ( get_cudnn_version, + get_nvtx_range_context, nvtx_range_pop, nvtx_range_push, get_device_compute_capability, @@ -62,6 +63,12 @@ _dpa_fp8_cs_o_in_f16 = os.getenv("NVTE_DPA_FP8CS_O_in_F16", "1") == "1" +def _nvtx_call(label, function, *args, **kwargs): + """Run one profiled operation without changing its synchronization semantics.""" + with get_nvtx_range_context(label): + return function(*args, **kwargs) + + def _reject_custom_recipe_under_cp(fp8, fp8_recipe): """Fail fast when CustomRecipe meets context-parallel FP8 attention. @@ -550,15 +557,21 @@ def flash_attn_a2a_communicate( a2a_inputs = [a2a_inputs] if not isinstance(a2a_inputs, list) else a2a_inputs a2a_outputs, a2a_reqs = [None] * len(a2a_inputs), [None] * len(a2a_inputs) _, _, head_dim = get_bsh_dims(qkv_format) + phase = "before_attention" if before_attn else "after_attention" if before_attn: for i in range(len(a2a_inputs) + 2): if 0 < i < len(a2a_inputs) + 1: - a2a_outputs[i - 1] = torch.empty_like(a2a_inputs[i - 1]) - a2a_reqs[i - 1] = torch.distributed.all_to_all_single( - a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True - ) + name = a2a_input_names[i - 1] + with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.launch"): + a2a_outputs[i - 1] = torch.empty_like(a2a_inputs[i - 1]) + a2a_reqs[i - 1] = torch.distributed.all_to_all_single( + a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True + ) if i > 1: - with torch.cuda.stream(cp_stream): + name = a2a_input_names[i - 2] + with torch.cuda.stream(cp_stream), get_nvtx_range_context( + f"transformer_engine.cp.a2a.{phase}.{name}.wait_and_reorder" + ): a2a_reqs[i - 2].wait() x = a2a_outputs[i - 2] if qkv_format in ["bshd", "sbhd", "bhsd"]: @@ -586,52 +599,61 @@ def flash_attn_a2a_communicate( ) if i < len(a2a_inputs): - x = a2a_inputs[i] - # [b, s, h, d] -> [b, s, cp, h//cp, d] - # [s, b, h, d] -> [s, b, cp, h//cp, d] - # [b, h, s, d] -> [b, cp, h//cp, s, d] - # [t, h, d] -> [t, cp, h//cp, d] - x = x.view( - *x.shape[:head_dim], - cp_size, - x.shape[head_dim] // cp_size, - *x.shape[head_dim + 1 :], - ) - # [b, s, cp, h//cp, d] -> [cp, b, s, h//cp, d] - # [s, b, cp, h//cp, d] -> [cp, s, b, h//cp, d] - # [b, cp, h//cp, s, d] -> [cp, b, h//cp, s, d] - # [t, cp, h//cp, d] -> [cp, t, h//cp, d] - a2a_inputs[i] = x.movedim(head_dim, 0).contiguous() + name = a2a_input_names[i] + with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.pack"): + x = a2a_inputs[i] + # [b, s, h, d] -> [b, s, cp, h//cp, d] + # [s, b, h, d] -> [s, b, cp, h//cp, d] + # [b, h, s, d] -> [b, cp, h//cp, s, d] + # [t, h, d] -> [t, cp, h//cp, d] + x = x.view( + *x.shape[:head_dim], + cp_size, + x.shape[head_dim] // cp_size, + *x.shape[head_dim + 1 :], + ) + # [b, s, cp, h//cp, d] -> [cp, b, s, h//cp, d] + # [s, b, cp, h//cp, d] -> [cp, s, b, h//cp, d] + # [b, cp, h//cp, s, d] -> [cp, b, h//cp, s, d] + # [t, cp, h//cp, d] -> [cp, t, h//cp, d] + a2a_inputs[i] = x.movedim(head_dim, 0).contiguous() else: for i in range(len(a2a_inputs) + 2): if 0 < i < len(a2a_inputs) + 1: - a2a_outputs[i - 1] = torch.empty_like(a2a_inputs[i - 1]) - a2a_reqs[i - 1] = torch.distributed.all_to_all_single( - a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True - ) - if i < len(a2a_inputs): - x = a2a_inputs[i] - if qkv_format in ["bshd", "sbhd", "bhsd"]: - # [b, cp*s, h//cp, d] -> [b, cp*2, s//2, h//cp, d] - # [cp*s, b, h//cp, d] -> [cp*2, s//2, b, h//cp, d] - # [b, h//cp, cp*s, d] -> [b, h//cp, cp*2, s//2, d] - x = x.view(*x.shape[:seq_dim], cp_size * 2, -1, *x.shape[(seq_dim + 1) :]) - # reorder the sequence chunks - a2a_inputs[i] = reorder_seq_chunks_for_a2a_after_attn( - x, chunk_ids_for_a2a, seq_dim, cp_size + name = a2a_input_names[i - 1] + with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.launch"): + a2a_outputs[i - 1] = torch.empty_like(a2a_inputs[i - 1]) + a2a_reqs[i - 1] = torch.distributed.all_to_all_single( + a2a_outputs[i - 1], a2a_inputs[i - 1], group=cp_group, async_op=True ) - else: # qkv_format == "thd" - cu_seqlens_padded = ( - cu_seqlens_q_padded - if a2a_input_names[i] in ["q", "out", "dout", "dq"] - else cu_seqlens_kv_padded - ) - # reorder the sequence chunks - x = thd_sequence_order_to_cp_rank_order(x, cu_seqlens_padded, cp_size) - # [cp*t, h//cp, d] -> [cp, t, h//cp, d] - a2a_inputs[i] = x.view(cp_size, -1, *x.shape[-2:]) + if i < len(a2a_inputs): + name = a2a_input_names[i] + with get_nvtx_range_context(f"transformer_engine.cp.a2a.{phase}.{name}.pack"): + x = a2a_inputs[i] + if qkv_format in ["bshd", "sbhd", "bhsd"]: + # [b, cp*s, h//cp, d] -> [b, cp*2, s//2, h//cp, d] + # [cp*s, b, h//cp, d] -> [cp*2, s//2, b, h//cp, d] + # [b, h//cp, cp*s, d] -> [b, h//cp, cp*2, s//2, d] + x = x.view(*x.shape[:seq_dim], cp_size * 2, -1, *x.shape[(seq_dim + 1) :]) + # reorder the sequence chunks + a2a_inputs[i] = reorder_seq_chunks_for_a2a_after_attn( + x, chunk_ids_for_a2a, seq_dim, cp_size + ) + else: # qkv_format == "thd" + cu_seqlens_padded = ( + cu_seqlens_q_padded + if a2a_input_names[i] in ["q", "out", "dout", "dq"] + else cu_seqlens_kv_padded + ) + # reorder the sequence chunks + x = thd_sequence_order_to_cp_rank_order(x, cu_seqlens_padded, cp_size) + # [cp*t, h//cp, d] -> [cp, t, h//cp, d] + a2a_inputs[i] = x.view(cp_size, -1, *x.shape[-2:]) if i > 1: - with torch.cuda.stream(cp_stream): + name = a2a_input_names[i - 2] + with torch.cuda.stream(cp_stream), get_nvtx_range_context( + f"transformer_engine.cp.a2a.{phase}.{name}.wait_and_reorder" + ): a2a_reqs[i - 2].wait() x = a2a_outputs[i - 2] # [cp, 2, b, s//2, h//cp, d] -> [2, b, s//2, cp, h//cp, d] @@ -1080,7 +1102,9 @@ def cp_p2p_fwd_fused_attn( fp8_meta_kwargs["s_quantizer"] = S_quantizer_per_step fp8_meta_kwargs["o_quantizer"] = O_quantizer_per_step - out_per_step, aux_ctx_tensors, *max_logit = fused_attn_fwd( + out_per_step, aux_ctx_tensors, *max_logit = _nvtx_call( + f"transformer_engine.cp.p2p.fwd.attention.{section}", + fused_attn_fwd, is_training, max_seqlen_q_, max_seqlen_kv_, @@ -1174,7 +1198,9 @@ def cp_p2p_fwd_flash_attn( cu_seqlens_q_ = cu_seqlens_q_padded // 2 if use_flash_attn_4: - fa_outputs = flash_attn_fwd( + fa_outputs = _nvtx_call( + f"transformer_engine.cp.p2p.fwd.attention.{section}", + flash_attn_fwd, q_part, k_part, v_part, @@ -1205,7 +1231,9 @@ def cp_p2p_fwd_flash_attn( seqused_q=seqused_q, seqused_k=seqused_k, ) - fa_outputs = flash_attn_fwd( + fa_outputs = _nvtx_call( + f"transformer_engine.cp.p2p.fwd.attention.{section}", + flash_attn_fwd, q_part, k_part, v_part, @@ -1402,7 +1430,9 @@ def cp_p2p_bwd_fused_attn( fp8_meta_kwargs["dp_quantizer"] = dP_quantizer_per_step fp8_meta_kwargs["dqkv_quantizer"] = dQKV_quantizer_per_step - dq, dk, dv, dbias, *_ = fused_attn_bwd( + dq, dk, dv, dbias, *_ = _nvtx_call( + f"transformer_engine.cp.p2p.bwd.attention.{section}", + fused_attn_bwd, max_seqlen_q_, max_seqlen_kv_, cu_seqlens_q_per_step[cp_size - step - 1], @@ -1551,7 +1581,9 @@ def cp_p2p_bwd_flash_attn( else: fa_backward_kwargs["causal"] = causal_ if use_flash_attn_4: - dq, dk, dv = flash_attn_bwd( + dq, dk, dv = _nvtx_call( + f"transformer_engine.cp.p2p.bwd.attention.{section}", + flash_attn_bwd, q_part, k_part, v_part, @@ -1561,7 +1593,9 @@ def cp_p2p_bwd_flash_attn( **fa_backward_kwargs, ) else: - flash_attn_bwd( + _nvtx_call( + f"transformer_engine.cp.p2p.bwd.attention.{section}", + flash_attn_bwd, dout_part, q_part, k_part, @@ -1927,22 +1961,32 @@ def forward( second_half_lse_seqlen = None for i in range(cp_size + 1): if i < cp_size: - with torch.cuda.stream(flash_attn_streams[i % 2]): + with torch.cuda.stream(flash_attn_streams[i % 2]), get_nvtx_range_context( + f"transformer_engine.cp.p2p.fwd.step_{i}.compute" + ): # wait until KV is received - for req in send_recv_reqs[(i + 1) % 2]: - req.wait() + with get_nvtx_range_context( + f"transformer_engine.cp.p2p.fwd.step_{i}.recv_wait" + ): + for req in send_recv_reqs[(i + 1) % 2]: + req.wait() if i < (cp_size - 1): - p2p_comm_buffers[(i + 1) % 2] = torch.empty_like(p2p_comm_buffers[i % 2]) - send_recv_reqs[i % 2] = flash_attn_p2p_communicate( - rank, - p2p_comm_buffers[i % 2], - send_dst, - p2p_comm_buffers[(i + 1) % 2], - recv_src, - cp_group, - batch_p2p_comm, - ) + with get_nvtx_range_context( + f"transformer_engine.cp.p2p.fwd.step_{i}.send_recv_launch" + ): + p2p_comm_buffers[(i + 1) % 2] = torch.empty_like( + p2p_comm_buffers[i % 2] + ) + send_recv_reqs[i % 2] = flash_attn_p2p_communicate( + rank, + p2p_comm_buffers[i % 2], + send_dst, + p2p_comm_buffers[(i + 1) % 2], + recv_src, + cp_group, + batch_p2p_comm, + ) kv_inputs[i % 2] = p2p_comm_buffers[i % 2] k_part = kv_inputs[i % 2][:k_numel].view(*k_shape) @@ -2147,7 +2191,9 @@ def forward( if i > 1: flash_attn_streams[(i - 1) % 2].wait_event(fwd_results_correction_done) - with torch.cuda.stream(flash_attn_streams[(i - 1) % 2]): + with torch.cuda.stream(flash_attn_streams[(i - 1) % 2]), get_nvtx_range_context( + f"transformer_engine.cp.p2p.fwd.step_{i - 1}.result_merge" + ): if use_fused_attention: # [b, h, sq, 1] -> [b, h, sq] or [t, h, 1] -> [t, h] softmax_lse_per_step[(i - 1) % 2].squeeze_(-1) @@ -2249,9 +2295,10 @@ def forward( torch.cuda.current_stream().wait_stream(flash_attn_streams[1]) if return_max_logit: - torch.distributed.all_reduce( - max_logit, op=torch.distributed.ReduceOp.MAX, group=cp_group - ) + with get_nvtx_range_context("transformer_engine.cp.p2p.fwd.max_logit_all_reduce"): + torch.distributed.all_reduce( + max_logit, op=torch.distributed.ReduceOp.MAX, group=cp_group + ) # Save the rank-local output for backward before restoring the A2A layout. out = out.view(post_a2a_o_shape) @@ -2699,14 +2746,18 @@ def backward(ctx, dout, *_args): send_recv_reqs = [] for i in range(cp_size): # wait until KV is received - for req in send_recv_reqs: - req.wait() + with get_nvtx_range_context(f"transformer_engine.cp.p2p.bwd.step_{i}.recv_wait"): + for req in send_recv_reqs: + req.wait() send_tensor = p2p_comm_buffers[i % 2] recv_tensor = p2p_comm_buffers[(i + 1) % 2] + exchange_label = f"transformer_engine.cp.p2p.bwd.step_{i}.gradient_exchange_launch" if ctx.fp8: if i < cp_size - 1: - send_recv_reqs = flash_attn_p2p_communicate( + send_recv_reqs = _nvtx_call( + exchange_label, + flash_attn_p2p_communicate, rank, send_tensor[0], send_dst, @@ -2716,7 +2767,9 @@ def backward(ctx, dout, *_args): batch_p2p_comm, ) else: - dkv_a2a_req = torch.distributed.all_to_all_single( + dkv_a2a_req = _nvtx_call( + exchange_label, + torch.distributed.all_to_all_single, dkv_send_buffer, dkv_recv_buffer, group=ctx.cp_group, @@ -2730,8 +2783,16 @@ def backward(ctx, dout, *_args): if i == (cp_size - 1): send_tensor = send_tensor[1] recv_tensor = recv_tensor[1] - send_recv_reqs = flash_attn_p2p_communicate( - rank, send_tensor, send_dst, recv_tensor, recv_src, ctx.cp_group, batch_p2p_comm + send_recv_reqs = _nvtx_call( + exchange_label, + flash_attn_p2p_communicate, + rank, + send_tensor, + send_dst, + recv_tensor, + recv_src, + ctx.cp_group, + batch_p2p_comm, ) kv = p2p_comm_buffers[i % 2][0] @@ -2956,8 +3017,9 @@ def backward(ctx, dout, *_args): attn_dbias_[..., 1, :, (2 * cp_size - idx - 1), :].copy_(dbias_[..., 1, :]) # wait until dKV is received - for req in send_recv_reqs: - req.wait() + with get_nvtx_range_context(f"transformer_engine.cp.p2p.bwd.step_{i}.dkv_recv_wait"): + for req in send_recv_reqs: + req.wait() # dkv correction if ctx.fp8 and ctx.fp8_recipe.delayed(): @@ -3464,24 +3526,50 @@ def forward( # AllGather K/V across CP ranks # gather along s or t: [s, b, h, d] -> [cp, s, b, h, d] or [t, h, d] -> [cp*t, h, d] - k_ag, _ = gather_along_first_dim(k, cp_group) - v_ag, _ = gather_along_first_dim(v, cp_group) + k_ag, _ = _nvtx_call( + "transformer_engine.cp.all_gather.fwd.k_gather", gather_along_first_dim, k, cp_group + ) + v_ag, _ = _nvtx_call( + "transformer_engine.cp.all_gather.fwd.v_gather", gather_along_first_dim, v, cp_group + ) if qkv_format == "thd": # [cp*t, h, d] -> reorder to sequence order -> [t_full, h, d] - k_ag = restore_thd_gathered_kv( - k_ag, cu_seqlens_kv_padded, cp_size, load_balancing_strategy + k_ag = _nvtx_call( + "transformer_engine.cp.all_gather.fwd.k_reorder", + restore_thd_gathered_kv, + k_ag, + cu_seqlens_kv_padded, + cp_size, + load_balancing_strategy, ) - v_ag = restore_thd_gathered_kv( - v_ag, cu_seqlens_kv_padded, cp_size, load_balancing_strategy + v_ag = _nvtx_call( + "transformer_engine.cp.all_gather.fwd.v_reorder", + restore_thd_gathered_kv, + v_ag, + cu_seqlens_kv_padded, + cp_size, + load_balancing_strategy, ) else: # [cp, s, b, h, d] -> [cp*2, s//2, b, h, d] k_ag = k_ag.view(2 * cp_size, k.shape[0] // 2, *k.shape[1:]) v_ag = v_ag.view(2 * cp_size, v.shape[0] // 2, *v.shape[1:]) chunk_ids_for_kv_ag = get_seq_chunk_ids_for_reordering_before_attn(cp_size, k.device) - k_ag = torch.index_select(k_ag, dim=0, index=chunk_ids_for_kv_ag) - v_ag = torch.index_select(v_ag, dim=0, index=chunk_ids_for_kv_ag) + k_ag = _nvtx_call( + "transformer_engine.cp.all_gather.fwd.k_reorder", + torch.index_select, + k_ag, + dim=0, + index=chunk_ids_for_kv_ag, + ) + v_ag = _nvtx_call( + "transformer_engine.cp.all_gather.fwd.v_reorder", + torch.index_select, + v_ag, + dim=0, + index=chunk_ids_for_kv_ag, + ) # [cp*2, s//2, b, h, d] -> [cp*s, b, h, d] k_ag = k_ag.view(-1, *k.shape[1:]) v_ag = v_ag.view(-1, *v.shape[1:]) @@ -3626,7 +3714,9 @@ def forward( # do not overlap. FusedAttention keeps the existing per-step overlap. if i > 0 and (use_flash_attn_3 or use_flash_attn_4): flash_attn_streams[i].wait_stream(flash_attn_streams[i - 1]) - with torch.cuda.stream(flash_attn_streams[i]): + with torch.cuda.stream(flash_attn_streams[i]), get_nvtx_range_context( + f"transformer_engine.cp.all_gather.fwd.step_{i}.compute" + ): new_qkv_layout = qkv_layout qkv_scale_inv_format = None if qkv_format in ["bshd", "sbhd"]: @@ -3711,7 +3801,9 @@ def forward( out_per_step[i], aux_ctx_tensors, *max_logit_, - ) = fused_attn_fwd( + ) = _nvtx_call( + f"transformer_engine.cp.all_gather.fwd.step_{i}.attention", + fused_attn_fwd, is_training, max_seqlen_q, max_seqlen_kv_, @@ -3772,7 +3864,9 @@ def forward( fa_forward_kwargs["window_size_left"] = window_size_per_step[i][0] fa_forward_kwargs["window_size_right"] = window_size_per_step[i][1] if use_flash_attn_4: - fa_outputs = flash_attn_fwd( + fa_outputs = _nvtx_call( + f"transformer_engine.cp.all_gather.fwd.step_{i}.attention", + flash_attn_fwd, q_part, k_part, v_part, @@ -3803,7 +3897,9 @@ def forward( seqused_q=seqused_q, seqused_k=seqused_k, ) - fa_outputs = flash_attn_fwd( + fa_outputs = _nvtx_call( + f"transformer_engine.cp.all_gather.fwd.step_{i}.attention", + flash_attn_fwd, q_part, k_part, v_part, @@ -3831,7 +3927,9 @@ def forward( if return_max_logit and i == 0: max_logit = torch.clone(max_logit_per_step[0]) if i > 0: - with torch.cuda.stream(flash_attn_streams[i - 1]): + with torch.cuda.stream(flash_attn_streams[i - 1]), get_nvtx_range_context( + f"transformer_engine.cp.all_gather.fwd.step_{i - 1}.result_merge" + ): if o_format == "bshd": out_f16[:, i - 1].copy_(out_per_step[i - 1]) elif o_format == "sbhd": @@ -3859,9 +3957,12 @@ def forward( # all reduce max_logit across ranks if return_max_logit: - torch.distributed.all_reduce( - max_logit, op=torch.distributed.ReduceOp.MAX, group=cp_group - ) + with get_nvtx_range_context( + "transformer_engine.cp.all_gather.fwd.max_logit_all_reduce" + ): + torch.distributed.all_reduce( + max_logit, op=torch.distributed.ReduceOp.MAX, group=cp_group + ) if qkv_format == "thd": out_f16 = out @@ -4110,18 +4211,38 @@ def backward(ctx, dout, *_args): dkv_update_done = torch.cuda.Event() # gather k and v along s or t: [s, b, h, d] -> [cp, s, b, h, d] or [t, h, d] -> [cp*t, h, d] - k_ag, _ = gather_along_first_dim(k, ctx.cp_group) - v_ag, _ = gather_along_first_dim(v, ctx.cp_group) + k_ag, _ = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.k_gather", + gather_along_first_dim, + k, + ctx.cp_group, + ) + v_ag, _ = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.v_gather", + gather_along_first_dim, + v, + ctx.cp_group, + ) if ctx.qkv_format == "thd": cu_seqlens_kv_padded = ctx.cu_seqlens_kv_padded thd_cu_seqlens_q_per_step = ctx.thd_cu_seqlens_q_per_step # [cp*t, h, d] -> reorder to sequence order - k_ag = restore_thd_gathered_kv( - k_ag, cu_seqlens_kv_padded, cp_size, ctx.load_balancing_strategy + k_ag = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.k_reorder", + restore_thd_gathered_kv, + k_ag, + cu_seqlens_kv_padded, + cp_size, + ctx.load_balancing_strategy, ) - v_ag = restore_thd_gathered_kv( - v_ag, cu_seqlens_kv_padded, cp_size, ctx.load_balancing_strategy + v_ag = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.v_reorder", + restore_thd_gathered_kv, + v_ag, + cu_seqlens_kv_padded, + cp_size, + ctx.load_balancing_strategy, ) thd_cu_seqlens_q_padded_per_step = ctx.thd_cu_seqlens_q_padded_per_step @@ -4131,8 +4252,20 @@ def backward(ctx, dout, *_args): v_ag = v_ag.view(2 * cp_size, v.shape[0] // 2, *v.shape[1:]) # select appropriate chunks for each rank chunk_ids_for_kv_ag = get_seq_chunk_ids_for_reordering_before_attn(cp_size, k.device) - k_ag = torch.index_select(k_ag, dim=0, index=chunk_ids_for_kv_ag) - v_ag = torch.index_select(v_ag, dim=0, index=chunk_ids_for_kv_ag) + k_ag = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.k_reorder", + torch.index_select, + k_ag, + dim=0, + index=chunk_ids_for_kv_ag, + ) + v_ag = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.v_reorder", + torch.index_select, + v_ag, + dim=0, + index=chunk_ids_for_kv_ag, + ) # flatten: [cp*2, s//2, b, h, d] -> [cp*s, b, h, d] k_ag = k_ag.view(-1, *k.shape[1:]) v_ag = v_ag.view(-1, *v.shape[1:]) @@ -4197,7 +4330,9 @@ def backward(ctx, dout, *_args): # per-step overlap. if i > 0 and (ctx.use_flash_attn_3 or ctx.use_flash_attn_4): flash_attn_streams[i].wait_stream(flash_attn_streams[i - 1]) - with torch.cuda.stream(flash_attn_streams[i]): + with torch.cuda.stream(flash_attn_streams[i]), get_nvtx_range_context( + f"transformer_engine.cp.all_gather.bwd.step_{i}.compute" + ): if ctx.qkv_format == "thd": # THD passes full Q/dout; per-step cu_seqlens select chunks. q_part = q @@ -4298,7 +4433,9 @@ def backward(ctx, dout, *_args): [(dout_part, ctx.dO_quantizer)], do_format, ) - dq_per_step[i], dk_per_step[i], dv_per_step[i], *_ = fused_attn_bwd( + dq_per_step[i], dk_per_step[i], dv_per_step[i], *_ = _nvtx_call( + f"transformer_engine.cp.all_gather.bwd.step_{i}.attention", + fused_attn_bwd, ctx.max_seqlen_q, max_seqlen_kv, cu_seqlens_q_, @@ -4422,7 +4559,9 @@ def backward(ctx, dout, *_args): dq_per_step[i], dk_per_step[i], dv_per_step[i], - ) = flash_attn_bwd( + ) = _nvtx_call( + f"transformer_engine.cp.all_gather.bwd.step_{i}.attention", + flash_attn_bwd, q_part, k_part, v_part, @@ -4432,7 +4571,9 @@ def backward(ctx, dout, *_args): **fa_backward_kwargs, ) else: - flash_attn_bwd( + _nvtx_call( + f"transformer_engine.cp.all_gather.bwd.step_{i}.attention", + flash_attn_bwd, dout_part, q_part, k_part, @@ -4445,7 +4586,9 @@ def backward(ctx, dout, *_args): if i > 0: # dq/dk/dv, dq_per_step/dk_per_step/dv_per_step: ctx.fwd_nominal_dtype - with torch.cuda.stream(flash_attn_streams[i - 1]): + with torch.cuda.stream(flash_attn_streams[i - 1]), get_nvtx_range_context( + f"transformer_engine.cp.all_gather.bwd.step_{i - 1}.gradient_merge" + ): if ctx.qkv_format == "thd": # dQ: copy every sequence's valid token range from this split's dQ. tex.thd_copy_valid_tokens_from_per_split_to_rank_local( @@ -4495,29 +4638,62 @@ def backward(ctx, dout, *_args): if ctx.qkv_format == "thd": # Reorder dK/dV from sequence order back to dual-chunk CP rank order, # then reduce-scatter across CP ranks. - dk = unrestore_thd_gathered_kv( - dk, cu_seqlens_kv_padded, cp_size, ctx.load_balancing_strategy + dk = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.dk_reorder", + unrestore_thd_gathered_kv, + dk, + cu_seqlens_kv_padded, + cp_size, + ctx.load_balancing_strategy, ) - dv = unrestore_thd_gathered_kv( - dv, cu_seqlens_kv_padded, cp_size, ctx.load_balancing_strategy + dv = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.dv_reorder", + unrestore_thd_gathered_kv, + dv, + cu_seqlens_kv_padded, + cp_size, + ctx.load_balancing_strategy, + ) + dk, _ = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.dk_reduce_scatter", + reduce_scatter_along_first_dim, + dk, + ctx.cp_group, + ) + dv, _ = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.dv_reduce_scatter", + reduce_scatter_along_first_dim, + dv, + ctx.cp_group, ) - dk, _ = reduce_scatter_along_first_dim(dk, ctx.cp_group) - dv, _ = reduce_scatter_along_first_dim(dv, ctx.cp_group) # dQ is already [t_rank, h, d], no reshape needed else: # split s:[cp*s, b, h, d] -> [cp*2, s//2, b, h, d] - dk = dk.view(2 * cp_size, -1, *dk.shape[-3:]) - dv = dv.view(2 * cp_size, -1, *dv.shape[-3:]) - # put back together the right chunks for each rank - chunk_ids_for_kv_ag = get_seq_chunk_ids_for_reordering_after_attn(cp_size, dk.device) - dk = torch.index_select(dk, dim=0, index=chunk_ids_for_kv_ag) - dv = torch.index_select(dv, dim=0, index=chunk_ids_for_kv_ag) - # flatten: [cp*2, s//2, b, h, d] -> [cp*s, b, h, d] - dk = dk.view(-1, *dk.shape[-3:]) - dv = dv.view(-1, *dv.shape[-3:]) + with get_nvtx_range_context("transformer_engine.cp.all_gather.bwd.dkv_reorder"): + dk = dk.view(2 * cp_size, -1, *dk.shape[-3:]) + dv = dv.view(2 * cp_size, -1, *dv.shape[-3:]) + # put back together the right chunks for each rank + chunk_ids_for_kv_ag = get_seq_chunk_ids_for_reordering_after_attn( + cp_size, dk.device + ) + dk = torch.index_select(dk, dim=0, index=chunk_ids_for_kv_ag) + dv = torch.index_select(dv, dim=0, index=chunk_ids_for_kv_ag) + # flatten: [cp*2, s//2, b, h, d] -> [cp*s, b, h, d] + dk = dk.view(-1, *dk.shape[-3:]) + dv = dv.view(-1, *dv.shape[-3:]) # reduce scatter: [cp*s, b, h, d] -> [s, b, h, d] - dk, _ = reduce_scatter_along_first_dim(dk, ctx.cp_group) - dv, _ = reduce_scatter_along_first_dim(dv, ctx.cp_group) + dk, _ = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.dk_reduce_scatter", + reduce_scatter_along_first_dim, + dk, + ctx.cp_group, + ) + dv, _ = _nvtx_call( + "transformer_engine.cp.all_gather.bwd.dv_reduce_scatter", + reduce_scatter_along_first_dim, + dv, + ctx.cp_group, + ) # reshape to original format: # dq: [b, 2, s//2, h, d] or [2, s//2, b, h, d] -> [b, s, h, d] or [s, b, h, d] @@ -4749,26 +4925,28 @@ def forward( # [s//cp, b, h, d] -> [s, b, h//cp, d] # [t//cp, h, d] -> [t, h//cp, d] chunk_ids_for_a2a = get_seq_chunk_ids_for_reordering_before_attn(cp_size, q.device) - q, k, v = flash_attn_a2a_communicate( - [q, k, v], - chunk_ids_for_a2a, - seq_dim_qkv, - cp_size, - cp_group, - cp_stream, - before_attn=True, - qkv_format=qkv_format, - cu_seqlens_q_padded=cu_seqlens_q_padded, - cu_seqlens_kv_padded=cu_seqlens_kv_padded, - a2a_input_names=["q", "k", "v"], - ) + with get_nvtx_range_context("transformer_engine.cp.a2a.fwd.qkv_exchange"): + q, k, v = flash_attn_a2a_communicate( + [q, k, v], + chunk_ids_for_a2a, + seq_dim_qkv, + cp_size, + cp_group, + cp_stream, + before_attn=True, + qkv_format=qkv_format, + cu_seqlens_q_padded=cu_seqlens_q_padded, + cu_seqlens_kv_padded=cu_seqlens_kv_padded, + a2a_input_names=["q", "k", "v"], + ) # softmax_offset: split h # [1, h, 1, 1] -> [1, h//cp, 1, 1] if softmax_type != "vanilla": - softmax_offset = flash_attn_a2a_communicate_softmax_offset( - softmax_offset, 1, cp_size, cp_group, cp_stream, True - ) + with get_nvtx_range_context("transformer_engine.cp.a2a.fwd.softmax_offset_exchange"): + softmax_offset = flash_attn_a2a_communicate_softmax_offset( + softmax_offset, 1, cp_size, cp_group, cp_stream, True + ) # _part: inputs to attention kernel and saved for backward # note: they have post a2a shapes @@ -4810,7 +4988,9 @@ def forward( Float8Tensor.make_like(x, data=y, dtype=fwd_nominal_dtype) for x, y in zip([q_fp8, k_fp8, v_fp8], [q_part, k_part, v_part]) ] - out_, aux_ctx_tensors, *max_logit = fused_attn_fwd( + out_, aux_ctx_tensors, *max_logit = _nvtx_call( + "transformer_engine.cp.a2a.fwd.attention", + fused_attn_fwd, is_training, max_seqlen_q, max_seqlen_kv, @@ -4861,7 +5041,9 @@ def forward( fa_cu_seqlens_q = cu_seqlens_q_padded fa_cu_seqlens_kv = cu_seqlens_kv_padded if use_flash_attn_4: - fa_outputs = flash_attn_fwd( + fa_outputs = _nvtx_call( + "transformer_engine.cp.a2a.fwd.attention", + flash_attn_fwd, q_part, k_part, v_part, @@ -4892,7 +5074,9 @@ def forward( seqused_q=seqused_q, seqused_k=seqused_k, ) - fa_outputs = flash_attn_fwd( + fa_outputs = _nvtx_call( + "transformer_engine.cp.a2a.fwd.attention", + flash_attn_fwd, q_part, k_part, v_part, @@ -4923,18 +5107,19 @@ def forward( if isinstance(out_, Float8TensorStorage): out_ = out_._data chunk_ids_for_a2a = get_seq_chunk_ids_for_reordering_after_attn(cp_size, out_.device) - out_ = flash_attn_a2a_communicate( - out_, - chunk_ids_for_a2a, - seq_dim_o, - cp_size, - cp_group, - cp_stream, - before_attn=False, - qkv_format=o_format, - cu_seqlens_q_padded=cu_seqlens_q_padded, - a2a_input_names=["out"], - ) + with get_nvtx_range_context("transformer_engine.cp.a2a.fwd.output_exchange"): + out_ = flash_attn_a2a_communicate( + out_, + chunk_ids_for_a2a, + seq_dim_o, + cp_size, + cp_group, + cp_stream, + before_attn=False, + qkv_format=o_format, + cu_seqlens_q_padded=cu_seqlens_q_padded, + a2a_input_names=["out"], + ) # [b*s//cp, h, d] -> [b, s//cp, h, d] # [s//cp*b, h, d] -> [s//cp, b, h, d] # [t//cp, h, d] -> [t//cp, h, d] @@ -4960,9 +5145,10 @@ def forward( # all gather max logit if return_max_logit: - max_logit = flash_attn_a2a_communicate_softmax_offset( - *max_logit, 0, cp_size, cp_group, cp_stream, False - ) + with get_nvtx_range_context("transformer_engine.cp.a2a.fwd.max_logit_exchange"): + max_logit = flash_attn_a2a_communicate_softmax_offset( + *max_logit, 0, cp_size, cp_group, cp_stream, False + ) ctx.qkv_layout = qkv_layout ctx.o_format = o_format @@ -5117,18 +5303,19 @@ def backward(ctx, dout, *_args): # [s//cp, b, h, d] -> [s, b, h//cp, d] # [t//cp, h, d] -> [t, h//cp, d] chunk_ids_for_a2a = get_seq_chunk_ids_for_reordering_before_attn(cp_size, dout.device) - dout = flash_attn_a2a_communicate( - dout, - chunk_ids_for_a2a, - seq_dim_do, - cp_size, - ctx.cp_group, - ctx.cp_stream, - before_attn=True, - qkv_format=ctx.o_format, - cu_seqlens_q_padded=cu_seqlens_q_padded, - a2a_input_names=["dout"], - ) + with get_nvtx_range_context("transformer_engine.cp.a2a.bwd.dout_exchange"): + dout = flash_attn_a2a_communicate( + dout, + chunk_ids_for_a2a, + seq_dim_do, + cp_size, + ctx.cp_group, + ctx.cp_stream, + before_attn=True, + qkv_format=ctx.o_format, + cu_seqlens_q_padded=cu_seqlens_q_padded, + a2a_input_names=["dout"], + ) flash_attn_bwd = None if not ctx.use_fused_attention: @@ -5198,7 +5385,9 @@ def backward(ctx, dout, *_args): [(dout, ctx.dO_quantizer)], do_format, ) - dq, dk, dv, *rest = fused_attn_bwd( + dq, dk, dv, *rest = _nvtx_call( + "transformer_engine.cp.a2a.bwd.attention", + fused_attn_bwd, ctx.max_seqlen_q, ctx.max_seqlen_kv, cu_seqlens_q, @@ -5293,7 +5482,9 @@ def backward(ctx, dout, *_args): fa_backward_kwargs["is_causal"] = causal if ctx.use_flash_attn_4: - dq, dk, dv = flash_attn_bwd( + dq, dk, dv = _nvtx_call( + "transformer_engine.cp.a2a.bwd.attention", + flash_attn_bwd, q, k, v, @@ -5303,7 +5494,9 @@ def backward(ctx, dout, *_args): **fa_backward_kwargs, ) else: - flash_attn_bwd( + _nvtx_call( + "transformer_engine.cp.a2a.bwd.attention", + flash_attn_bwd, dout, q, k, @@ -5327,19 +5520,20 @@ def backward(ctx, dout, *_args): # [s//cp, b, h, d] -> [s, b, h//cp, d] # [t//cp, h, d] -> [t, h//cp, d] chunk_ids_for_a2a = get_seq_chunk_ids_for_reordering_after_attn(cp_size, dq.device) - dq, dk, dv = flash_attn_a2a_communicate( - [dq, dk, dv], - chunk_ids_for_a2a, - seq_dim_dqkv, - cp_size, - ctx.cp_group, - ctx.cp_stream, - before_attn=False, - qkv_format=ctx.dqkv_format, - cu_seqlens_q_padded=cu_seqlens_q_padded, - cu_seqlens_kv_padded=cu_seqlens_kv_padded, - a2a_input_names=["dq", "dk", "dv"], - ) + with get_nvtx_range_context("transformer_engine.cp.a2a.bwd.dqkv_exchange"): + dq, dk, dv = flash_attn_a2a_communicate( + [dq, dk, dv], + chunk_ids_for_a2a, + seq_dim_dqkv, + cp_size, + ctx.cp_group, + ctx.cp_stream, + before_attn=False, + qkv_format=ctx.dqkv_format, + cu_seqlens_q_padded=cu_seqlens_q_padded, + cu_seqlens_kv_padded=cu_seqlens_kv_padded, + a2a_input_names=["dq", "dk", "dv"], + ) dq, dk, dv = [ x.view(y) for x, y in zip([dq, dk, dv], [ctx.orig_q_shape, ctx.orig_k_shape, ctx.orig_v_shape]) @@ -5353,9 +5547,12 @@ def backward(ctx, dout, *_args): d_bias = rest[0] if ctx.softmax_type != "vanilla": d_softmax_offset = rest[1] - d_softmax_offset = flash_attn_a2a_communicate_softmax_offset( - d_softmax_offset, 1, cp_size, ctx.cp_group, ctx.cp_stream, False - ) + with get_nvtx_range_context( + "transformer_engine.cp.a2a.bwd.softmax_offset_exchange" + ): + d_softmax_offset = flash_attn_a2a_communicate_softmax_offset( + d_softmax_offset, 1, cp_size, ctx.cp_group, ctx.cp_stream, False + ) # convert dq, dk, dv to appropriate types if ctx.fp8: