Skip to content

fix(pathfinder): find cuDNN in the Windows Arm64 wheel - #3489

Open
prewaskar-nv wants to merge 5 commits into
NVIDIA:mainfrom
prewaskar-nv:pathfinder-cudnn-win-arm64-wheel
Open

prewaskar-nv wants to merge 5 commits into
NVIDIA:mainfrom
prewaskar-nv:pathfinder-cudnn-win-arm64-wheel

Conversation

@prewaskar-nv

Copy link
Copy Markdown

Description

The nvidia-cudnn-cu13 win_arm64 wheel installs cudnn64_9.dll and its sub-libraries in nvidia/cudnn/bin, the same directory as the win_amd64 wheel. The cuDNN descriptor lists that site-packages directory for x64 only (WindowsSearchDirs.x64_only("nvidia/cudnn/bin")), so on Windows Arm64 load_nvidia_dynamic_lib("cudnn") skips the installed wheel. The only Arm64 route left is CUDNN_PATH/bin/arm64, which matches the standalone archive but not the wheel.

This lists nvidia/cudnn/bin for Arm64 as well. Conda, CUDA_PATH, and Program Files stay x64-only on Arm64, as #2680 left them.

Wheel layout, from the file listings of the wheels on PyPI:

Wheel DLL directory
nvidia_cudnn_cu13-9.28.0.42-py3-none-win_arm64.whl nvidia/cudnn/bin (10 DLLs)
nvidia_cudnn_cu13-9.28.1.4-py3-none-win_arm64.whl nvidia/cudnn/bin (10 DLLs)
nvidia_cudnn_cu13-9.28.1.4-py3-none-win_amd64.whl nvidia/cudnn/bin (10 DLLs)

Found while moving PyTorch's Windows-on-Arm builds to the NVIDIA wheels. torch loads cuDNN through cuda-pathfinder at import (pytorch/pytorch#202589) and preloads the other cudnn*.dll from the directory pathfinder returns (pytorch/pytorch#203663), so on Windows Arm64 it can't use the cuDNN wheel today.

Changes

  • descriptor_catalog.py: add nvidia/cudnn/bin to the Arm64 site_packages_windows entry, and update the comment on which Arm64 layouts are verified.
  • Tests:
    • test_cudnn_metadata_matches_supported_layouts and test_cudnn_arm64_archive_layout_is_not_assumed_for_other_roots pinned the old x64-only value; both now expect nvidia/cudnn/bin on Arm64.
    • New test_found_windows_arm64_cudnn_wheel: the site-packages step finds nvidia/cudnn/bin/cudnn64_9.dll on Arm64.
  • Release notes: start 1.9.1-notes.rst and add 1.9.1 to nv-versions.json, as Add pathfinder support for cupti64_2026.1.2.dll (CTK 13.2.2) #3279 did for 1.8.5.

Validation

  • pytest tests/ from cuda_pathfinder/ on Windows x64, Python 3.11: 1596 passed, 22 skipped.
  • Against the old catalog, the three cuDNN tests above fail; with this change they pass.
  • ruff check and ruff format --check (0.17.3, the pre-commit pin) pass on the changed files.
  • Not run: a real load on a Windows Arm64 host with the wheel installed. I don't have an Arm64 machine for this test.

AI Assistance Disclosure

This change and description were prepared with AI assistance (Cursor agent, Claude Opus 5.5). The new test carries @pytest.mark.agent_authored(model="claude-opus-5.5").

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

The nvidia-cudnn-cu13 win_arm64 wheel installs cudnn64_9.dll and its
sub-libraries in nvidia/cudnn/bin, the same directory as the win_amd64
wheel (checked for 9.28.0.42 and 9.28.1.4). The cuDNN descriptor listed
that site-packages directory for x64 only, so on Windows Arm64
load_nvidia_dynamic_lib("cudnn") skipped the wheel and could only find
cuDNN through CUDNN_PATH/bin/arm64.

List nvidia/cudnn/bin for arm64 as well. Conda, CUDA_PATH, and Program
Files stay x64-only on Windows Arm64.

Signed-off-by: Pawan Rewaskar <273072399+prewaskar-nv@users.noreply.github.com>
@github-actions github-actions Bot added the cuda.pathfinder Everything related to the cuda.pathfinder module label Nov 18, 2026
@prewaskar-nv
prewaskar-nv force-pushed the pathfinder-cudnn-win-arm64-wheel branch from 933110f to ff3775b Compare November 18, 2026 09:14
Signed-off-by: Pawan Rewaskar <273072399+prewaskar-nv@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • New Features
    • Windows ARM64 systems can discover cuDNN installed through the nvidia-cudnn-cu13 wheel.
    • cuDNN discovery checks DLL architecture and continues searching when a candidate is incompatible, allowing it to find a compatible installation in another supported location.
    • On Windows ARM64, conda, CUDA_PATH, and Program Files installations remain x64-only.
  • Documentation
    • Added 1.9.1 release notes describing Windows ARM64 cuDNN discovery and its installation limitations.

The cuDNN descriptor now searches Windows ARM64 wheel directories and checks candidate DLL architecture. If a wheel DLL is incompatible, the search can continue to a compatible installation. Tests and 1.9.1 release materials reflect the updated behavior.

Changes: Windows ARM64 cuDNN Wheel Discovery

Layer / File(s) Summary
Configure and verify cuDNN discovery
cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py, cuda_pathfinder/tests/test_descriptor_catalog.py, cuda_pathfinder/tests/test_search_steps.py, cuda_pathfinder/tests/test_load_nvidia_dynamic_lib_using_mocker.py
The cuDNN descriptor configures Windows wheel paths for x64 and ARM64 and checks candidate DLL architecture. Tests cover architecture-matched wheel selection, fallback to CUDNN_PATH, and search directory metadata.
Document version 1.9.1
cuda_pathfinder/docs/nv-versions.json, cuda_pathfinder/docs/source/release/1.9.1-notes.rst
The version list and release notes add version 1.9.1. The notes describe Windows ARM64 wheel discovery and state that Conda, CUDA_PATH, and Program Files installs remain x64-only on Windows ARM64.

Suggested labels: bug

Suggested reviewers: dekhtiarjonathan, rwgk

Change: Bug fix

Merge Risk: Minimal · up to f2bac

Windows Arm64 cuDNN discovery now finds the wheel and skips DLLs built for another architecture, so the search can continue to a compatible installation. No concrete merge-blocking risk remains.

Security architecture risk: Minimal · up to f2bac

The ARM64 wheel uses the existing wheel-first loading policy, while incompatible cuDNN DLLs are skipped before selection. No introduced or worsened material security, ownership or failure-containment risk was identified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — A substituted matching-architecture DLL in a selected installation directory would execute with the calling Python process's existing identity and access. The ARM64 wheel adds another eligible native-code location within configured site-packages roots; it does not grant a separate identity. Loading cuDNN also registers its directory in process-global DLL search state.

Trust Boundaries and Controls

  • observed — Wheel discovery uses system or virtual-environment site-packages and includes user-site directories only when Python enables them. This directory-selection policy is unchanged. The new PE filter applies to explicit disk discovery, while already-loaded modules and native system search continue through their existing paths without this Python-level check.

Resilience and Maintainability Implications

  • observed — Architecture rejection occurs before native loading or DLL-directory registration and publishes no loaded result. Once a candidate is accepted, existing loading is not transactional: dependencies may already be loaded, and directory registration plus PATH mutation precede the native load and remain if it fails. Successful results are cached. These lifecycle limitations predate the PR; the ARM64 wheel inherits them rather than introducing a new cleanup guarantee.
Finishing Touches

Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d7a01a3 and f2bacc7.

Files selected for processing: 5

  • cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py
  • cuda_pathfinder/docs/source/release/1.9.1-notes.rst
  • cuda_pathfinder/tests/test_descriptor_catalog.py
  • cuda_pathfinder/tests/test_load_nvidia_dynamic_lib_using_mocker.py
  • cuda_pathfinder/tests/test_search_steps.py
Files skipped from review as they are similar to previous changes: 1

  • cuda_pathfinder/docs/source/release/1.9.1-notes.rst

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 320532e and d7a01a3.

Files selected for processing: 5

  • cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py
  • cuda_pathfinder/docs/nv-versions.json
  • cuda_pathfinder/docs/source/release/1.9.1-notes.rst
  • cuda_pathfinder/tests/test_descriptor_catalog.py
  • cuda_pathfinder/tests/test_search_steps.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@github-actions

Copy link
Copy Markdown
Contributor

@prewaskar-nv

Copy link
Copy Markdown
Author

@rwgk @juenglin, could one of you review this when you have a moment? It's a one-entry catalog change: the nvidia-cudnn-cu13 win_arm64 wheel installs cudnn64_9.dll in nvidia/cudnn/bin, which pathfinder only searches on x64. This is what currently stops PyTorch's Windows-on-Arm builds from loading the cuDNN wheel through pathfinder.

The "PR has assignee, labels, and milestone" check needs an assignee and a milestone (probably cuda.pathfinder next). I don't have permission to set either, so could you add them? Apart from that check, CI has passed so far; some jobs are still running.

On CodeRabbit's suggestion to set requires_windows_binary_arch_check=True for cuDNN: the win_arm64 and win_amd64 wheels use the same nvidia/cudnn/bin path, so the check only matters if an x64 wheel ends up in an Arm64 environment. I'm happy to add it, as nvvm does, if you'd like the extra guard.

Thanks!

rwgk added 3 commits November 18, 2026 12:50
— codex GPT-6 on behalf of Ralf —

Check the PE architecture before selecting cuDNN from a shared wheel
location so discovery can continue to a compatible installation.

Cover matching x64 and Arm64 wheels and fallback from an x64 wheel to an
Arm64 archive. Update synthetic DLL fixtures for architecture validation.
— codex GPT-6 on behalf of Ralf —

Hide installed wheels from the archive loader test so site-packages
cannot override its temporary CUDNN_PATH installation.
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rwgk

rwgk commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thanks @prewaskar-nv for sending this PR! I added two commits to fix codex findings:

  • c658d92 — checks the cuDNN DLL’s architecture so an incompatible wheel does not block fallback to a compatible installation.

  • 5e5a5ce — isolates the Arm64 archive-loader test from installed wheels, fixing the Windows CI failures caused by site-packages taking precedence over the test archive.

@rwgk
rwgk self-requested a review November 18, 2026 21:37
@rwgk rwgk added this to the cuda.pathfinder next milestone Nov 18, 2026
@rwgk rwgk added bug Something isn't working P0 High priority - Must do! labels Nov 18, 2026
@rwgk

rwgk commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/ok to test f2bacc7

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@rwgk
rwgk enabled auto-merge (squash) November 18, 2026 21:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.pathfinder Everything related to the cuda.pathfinder module P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants