Add cuDNN discovery and NCCL header support - #2680
Conversation
github-actions
Bot
added
the
cuda.pathfinder
There was a problem hiding this comment.
Pull request overview
This pull request extends cuda_pathfinder’s component discovery catalog to cover cuDNN (shared libraries + headers) and NCCL headers, and wires those additions into the test/dependency setup used for validating wheel and conda layouts.
Changes:
- Add cuDNN dynamic-library descriptor metadata for Linux and Windows, plus a catalog test to validate expected wheel layout metadata.
- Add header descriptor metadata for cuDNN (Linux/Windows) and NCCL (Linux-only), plus a test to validate header metadata against expected wheel layouts.
- Update pathfinder conda test-environment helpers and
cuda_pathfindertest dependency groups to include cuDNN (and NCCL on Linux).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
toolshed/conda_create_for_pathfinder_testing.sh |
Add cudnn and nccl to the Linux conda-based test environment packages. |
toolshed/conda_create_for_pathfinder_testing.ps1 |
Add cudnn to the Windows conda-based test environment packages. |
cuda_pathfinder/tests/test_find_nvidia_headers.py |
Extend non-CTK distribution metadata patterns for cuDNN/NCCL and add a header descriptor layout test. |
cuda_pathfinder/tests/test_descriptor_catalog.py |
Add a descriptor-catalog test validating cuDNN wheel layout metadata. |
cuda_pathfinder/pyproject.toml |
Add nvidia-cudnn-cu12 / nvidia-cudnn-cu13 to the CUDA test dependency groups. |
cuda_pathfinder/cuda/pathfinder/_headers/header_descriptor_catalog.py |
Add header descriptor specs for cudnn and nccl. |
cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py |
Add a cuDNN dynamic-library descriptor spec for discovery on Linux/Windows. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- cuDNN: added dynamic-library loading and header discovery. - NCCL: dynamic-library loading already existed; added the missing header discovery.
19a6448 to
3d1eab9
Compare
|
/ok to test ddaa78f |
|
@DEKHTIARJonathan I pushed four commits (one of them adding WoA support) and triggered the CI here. I'll do the WoA testing on the internal repo. |
|
I forgot to add: I didn't fully review myself yet. |
|
|
/ok to test |
@rwgk, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/ |
|
/ok to test 39f25a0 |
|
codex gpt-5.6-sol ultra PR 2680: first successful public CI log analysisAnalyzed against public CI run 32522149396, commit 39f25a0 ( Executive conclusionThe public run is fully green: 100 jobs succeeded, two intentionally gated CUDA Core API checks were skipped, no job failed or was cancelled, and the final aggregate status job succeeded. The run provides strong, genuinely unmocked integration evidence for the main Pathfinder contract introduced or affected by PR 2680:
This is sufficient real integration coverage for the pip-wheel discovery and native-load claims on the three public platforms above. It is not sufficient real evidence for Windows on ARM, standalone My merge-readiness judgment is therefore:
MethodI treated a test as "real" only when it used an installed NVIDIA artifact and the production discovery or native-loading code without replacing that operation with a mock. I used three evidence levels:
The audit used the completed GitHub Actions metadata and all 65 Linux/Windows GPU test logs, then cross-checked the relevant workflow and test implementation at the run SHA. Representative job links are included below. Run overview
The 24 wheel-build jobs comprise eight each for Linux x86_64, Linux AArch64, and Windows x64. They cover Python 3.10 through 3.15, including selected 3.14t and 3.15t free-threaded builds. GPU test matrixEvery one of these 65 jobs successfully completed all three relevant steps:
The test jobs span Python 3.10, 3.11, 3.12, 3.13, 3.14, 3.14t, 3.15, and 3.15t where selected by the platform matrix. Windows additionally spans MCDM, TCC, and WDDM driver modes. An important qualification: Why the dynamic-library test is genuinely realThe relevant test is data-driven over the descriptor-derived supported-library tables in For each library,
Those assertions are in Strict mode is also meaningful. The expected-failure helper at Real Windows x64 evidenceAll 21 Windows GPU test logs were checked. Every strict run reported:
Representative CUDA 12 evidence is job 96899599802, Python 3.10 / CUDA 12.9 / RTX 2080 WDDM. Its first permissive phase reports cuDNN, cublasLt, and NVRTC as not found. The dependency step then installs Representative CUDA 13 evidence is job 96899599699, Python 3.14t / CUDA 13.3 / A100 MCDM. It installs Normal Windows jobs report Real Linux evidenceAll 44 Linux GPU test logs were checked: 26 x86_64 and 18 AArch64. Every log showed the same useful before/after transition:
This transition is particularly good evidence: it rules out a pre-existing system cuDNN or NCCL installation masking an error in wheel discovery. Across those 44 jobs:
Representative jobs:
The 44 strict Linux repetitions total 65,128 passed test instances and 212 expected skips. Those are matrix repetitions, not 65,128 unique tests; the value is breadth across environments rather than independent test-case count. Real header evidenceThe relevant data-driven test is
The cuDNN distribution matcher is deliberately limited to backend packages ( This is strong evidence for the actual header-wheel layouts on Windows x64, Linux x86_64, and Linux AArch64. It does not compile a translation unit or validate transitive includes. Dependency-loading evidenceThe cuDNN descriptor declares required The strict environments contain real cublasLt and NVRTC wheels, and every platform/family log shows both can be loaded as real native libraries. Therefore the run proves:
The per-cuDNN child payload does not emit a nested trace for each dependency. The source call graph guarantees the attempt, while the separate real load records establish artifact availability. The optional-NVRTC-absent case remains callback-based unit coverage rather than a real-binary integration scenario. Coverage of the CI-failure fixThe previous public run failed on two Windows-host path-string assumptions. In representative Windows job 96899599699, both corrected tests pass in both Pathfinder phases:
The first now exercises the host's real The two recurring Pathfinder warnings in representative jobs are pytest configuration/marker warnings for Coverage map for PR 2680 behavior
Residual risks and recommended follow-ups1. Native Windows ARM64 remains the largest gapPublic CI has no Windows ARM64 job. The WoA orchestration test at Minimum high-value follow-up: on a native WoA runner, provision a pinned/checksummed official ARM64 cuDNN archive plus compatible cublasLt, set 2. Standalone and system installation roots are not real-testedNo public job uses a real standalone cuDNN archive, a real Program Files installation, Linux cuDNN DEB/RPM headers, a real Recommended follow-ups, in priority order:
3. Native attachment is not product functionalityThe real dynamic test proves that the top-level cuDNN/NCCL library can be attached and that Pathfinder reports the right file. It does not resolve a product symbol, query a version, create a cuDNN handle, or run a convolution/graph operation. In particular, it may not force every component that cuDNN loads lazily. A lightweight 4. Version breadth is environmental, not contractualThe requirements correctly constrain cuDNN to major 9, but the resolver selected only Final assessmentThis is a high-quality green run for a discovery/loading library. The most important paths are not merely mocked: the CI installs the built Pathfinder wheel and current NVIDIA wheels, observes the products absent before installation, discovers real headers afterward, and attaches real libraries through fresh-process native loader calls under strict failure semantics. Coverage is broad across 65 GPU environments and both CUDA-major package families. Accordingly, the public CI is sufficient to support merging the Windows x64 and Linux wheel behavior in PR 2680 and to demonstrate that the two Windows portability regressions are fixed. The result should not be summarized as "all supported layouts and architectures are real-tested." Windows ARM64, standalone/system roots, conda, optional-NVRTC absence, header compilation, and post-load product behavior remain explicitly outside the real integration evidence. A native WoA archive smoke is the one follow-up most likely to change confidence materially. |
kkraus14
left a comment
kkraus14
left a comment
There was a problem hiding this comment.
Non-blocking follow-ups from a local review of the current head. CI and the local suite are green; the inline comments cover forward-compatible Windows DLL matching, DLL-directory registration for an already-loaded standalone cuDNN, and Linux product-root dynamic discovery. I omitted the native Windows ARM64 wheel-dependency point because public CUDA 13.4/WoA packaging is still pending.
| desc: LibDescriptor, | ||
| target_arch: str | None = None, | ||
| ) -> str | None: | ||
| for dll_basename in reversed(cast(tuple[str, ...], desc.windows_dlls)): |
There was a problem hiding this comment.
Non-blocking: making every Windows filesystem search exact fixes the cuDNN sidecar ambiguity, but it also removes the previous forward-compatible wildcard lookup for versioned DLL names. For example, the catalog currently ends at cupti64_2026.3.0.dll, so a newer CUPTI DLL in a wheel, conda env, or toolkit root will be ignored until Pathfinder is updated. Could exact-name selection be an opt-in descriptor policy (enabled for cuDNN), while retaining wildcard discovery for descriptors such as CUPTI?
| arm64=("bin/arm64",), | ||
| ), | ||
| program_files_root_globs_windows=WindowsSearchDirs.x64_only("NVIDIA/CUDNN/v9.*"), | ||
| requires_add_dll_directory=True, |
There was a problem hiding this comment.
Non-blocking edge case: this side effect is skipped when cudnn64_9.dll was already loaded from a late root such as CUDNN_PATH or Program Files. _load_lib_no_cache() passes find is not None to the already-loaded check, but find contains only the early wheel/conda result; the function then returns before late-root discovery. That can leave cuDNN's lazily loaded component DLLs undiscoverable. Could the already-loaded path register its resolved module directory whenever requires_add_dll_directory is set, independent of the early-find boolean, with a regression test for a preloaded standalone install?
| return cast(tuple[str, ...], desc.anchor_rel_dirs_linux) | ||
|
|
||
| def install_root_env_vars(self, _desc: LibDescriptor) -> tuple[str, ...]: | ||
| return () |
There was a problem hiding this comment.
Non-blocking: these no-op Linux hooks make the new product-root dynamic search Windows-only. Header discovery accepts CUDNN_PATH and NCCL_HOME, but dynamic discovery will not inspect locations such as $CUDNN_PATH/lib{,64} or $NCCL_HOME/{lib,build/lib} unless users also configure the OS loader path. Could we add Linux product-root metadata/relative directories and focused tests so header and library discovery stay aligned?
|
Thanks @kkraus14! I still haven't fully reviewed myself. I'm working on making the internal CI more complete. I have a fully successful internal CI run, but (codex analysis):
I believe it's fixable; and it's a good value, long-term. |
PR 2680: final ctk-next CI analysis with real WoA cuDNN coverageAnalyzed against the final green CI for ctk-next PR 537, which mirrors the product changes in public PR 2680 and adds ctk-next-only provisioning and verification for native Windows ARM64. Executive conclusionThe intended ctk-next matrix is fully green. Four platform workflows ran on Linux x86_64, Linux AArch64, Windows x64, and native Windows ARM64. Each completed its source/runbook job, wheel-build job, and isolated installed-wheel test job: 12/12 active platform jobs succeeded. Bandit also succeeded. The complete check rollup contains 13 successes and 17 checks intentionally skipped by branch or policy routing; there were no failures or cancellations. This result complements the already-posted public CI analysis for PR 2680:
MethodI used the same evidence standard as the public report:
For the new WoA conclusion, only the first level is used as proof. The dedicated strict tests and semantic smoke use the production Pathfinder public APIs, real ARM64 PE files, the native Windows loader, a real driver, and a real GB10 GPU. Their relevant discovery and loading operations are not mocked. I inspected the final check inventory, all four completed platform workflows, both WoA test-bearing job logs, and the exact workflow and verification source at the tested ctk-next head. Run inventoryAll four active platform workflows ran at the same final ctk-next PR 537 head:
Bandit also passed. Across the final PR check rollup, 13 checks succeeded and 17 were skipped. The skipped generic CI, security, labeling, and PR-policy checks are expected for this ctk-next ref-routing arrangement and should not be counted as test coverage. The four three-job platform workflows are the active matrix. Matrix and Pathfinder resultsAll platform jobs use CUDA 13.4.1, kitpick 011, and Python 3.13. Linux test-bearing jobs run in Ubuntu 24.04 containers with driver 615.62. Windows reports Python 3.13.14 with the native architecture for each runner. The full Pathfinder suite ran twice on every platform: once from the editable/source runbook installation and once from the built wheel in the isolated wheel-test environment.
The eight full-suite executions total 11,714 passed test instances and 48 skips. These totals reflect repeated source/wheel and platform coverage, not 11,714 unique tests. In addition, each WoA test-bearing job ran four dedicated strict real-artifact Pathfinder checks. Those eight additional test instances all passed. Each job then ran the semantic native cuDNN verification script successfully. The wheel jobs create a fresh virtual environment, install the locally built Pathfinder, bindings, and core wheels, move the checkout package trees aside, and fail if What ctk-next CI proves with real artifactsCUDA 13.4 and production Pathfinder loadingThe logs contain broad production-loader evidence against the real CUDA 13.4 kitpick:
This directly exercises the shared install-root/search refactoring under CUDA 13.4 and shows that the new cuDNN metadata and dependency handling do not regress established CUDA library discovery on either Windows architecture. Native Windows ARM64The WoA workflow is genuinely native, not x64 emulation with a patched platform string:
The full Pathfinder suite passes twice on this native host. Architecture detection, descriptor invariants, Windows path handling, source packaging, and the installed wheel therefore all execute successfully under real ARM64 Python and Windows filesystem semantics. Real WoA cuDNN, cuBLASLt, and NVRTC coverageArtifact provisioningThe successful source/runbook job had a cuDNN cache miss. It therefore exercised the complete cold path:
This matters because it proves discovery through the new descriptor-driven The CUDA 13.4 mini-toolkit contains real ARM64 The installed-wheel job restored the checksum-keyed cuDNN cache produced by the source job and re-ran the required-file and exact-DLL-set validation. The two jobs therefore cover both the cold download/extraction path and the subsequent cache-restore path. Dedicated strict Pathfinder checksBoth the source/runbook environment and the isolated installed-wheel environment set: Each then runs four targeted cases, all of which pass:
The dynamic-library cases run production Pathfinder in fresh subprocesses and use the native Windows loader. They do not mock the search context, dependency callback, loader, or DLL. The logged absolute paths are the intended artifacts: The cuDNN result cannot succeed merely by finding the shim. Its descriptor first loads required cuBLASLt and attempts optional NVRTC, registers the cuDNN directory for native dependency resolution, and then attaches the real ARM64 cuDNN DLL. Semantic native API smokeAfter the strict pytest cases, both environments run a stronger direct verification through the production Pathfinder APIs. It proves all of the following:
This goes beyond a top-level Source and installed-wheel repetitionThe entire strict and semantic sequence succeeds twice:
The installed-wheel job additionally runs the broader native package suites successfully. Its aggregate result, including the four targeted Pathfinder checks, is 6,265 passed, 414 skipped, two expected xfails, and zero failures, plus 29 successful subtests. This is useful corroboration that the real cuDNN path coexists with the complete native wheel stack rather than succeeding only in a narrowly constructed loader process. Combined public and ctk-next coverageThe two runs now cover complementary axes, including the intersection that was missing from the first ctk-next report:
The most accurate single-sentence combined summary is:
|
|
/ok to test 6dbbe8d |
|
/ok to test e5c91aa |
|
/ok to test 3319fc3 |
|
/ok to test 30a905ad0d4c4fcce383d673a562e15aadd0bd48 |
@rwgk, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test 30a905a |
Description
No linked issue; this change adds requested cuda-pathfinder support for cuDNN and NCCL.
Checklist