Windows platform support bringup

[ScottTodd]

Overview

We're aiming to support as much of the toolkit as possible on Windows.

• Our build target is "native" Windows (not WSL 1 or WSL 2), compiling using standard build tools (e.g. MSVC).
• We should support local builds, have regular CI builds (GitHub Actions), and eventually be able to distribute artifacts.
• We should also have any appropriate documentation, developer guides, etc. needed.

Current status

• Documentation for building and a list of currently supported subprojects is at https://github.com/ROCm/TheRock/blob/main/docs/development/windows_support.md.
  • The 'base' and 'compiler' components build fully from public sources
  • The 'core' component can build from public static libraries built from private sources (https://github.com/nod-ai/amdgpu-windows-interop).
  • The 'math libraries' components all build. Some tests have been validated
  • MIOpen builds
• The build_windows_packages.yml workflow runs as part of ci.yml on every commit. The workflow builds some supported subprojects and publishes all built artifacts to S3.
  • See also [Windows] Enable all subproject builds in CI #543
• The release_windows_packages.yml workflow runs nightly and publishes tarballs, rocm python packages, and torch packages to nightly releases.
  • See also [Windows] Build and upload nightly releases #542 and https://github.com/ROCm/TheRock/blob/main/RELEASES.md
• PyTorch support is in preview: [Windows] Support building PyTorch with ROCm support #589

Initial experiments 2025-01-30

Expand to see details

Setup notes

• I'm on Windows 11 version 23H2, OS build 22631.4751
• I enabled Windows developer mode and granted my user account the ability to create symlinks
• I created a Dev Drive by following https://learn.microsoft.com/en-us/windows/dev-drive/ (500GB, shared with other projects)
• I installed standard software development tools and configured developer options:

  • git from https://git-scm.com/downloads (latest version 2.45.1) then enabled symlinks with

    git config --global core.symlinks true
    

  • Build Tools for Visual Studio 2022 from https://visualstudio.microsoft.com/downloads/ (latest version 17.12.4)

  • CMake from https://cmake.org/download/ (latest, version 3.31.5)

  • Ninja as a CMake generator from https://ninja-build.org/ (latest, version 1.12.1)

  • ccache from https://ccache.dev/ (latest, version 4.10.2)

Fetching sources

After cloning TheRock, I tried to checkout sources using python ./build_tools/fetch_sources.py. That needed the repo tool (https://gerrit.googlesource.com/git-repo/+/HEAD/README.md, docs at https://source.android.com/docs/setup/reference/repo), which is not distributed on Windows, so I downloaded the script manually. I did then have to edit build_tools/fetch_sources.py to exec python D:/path/to/repo instead of just repo. We can teach that script how to download the file on its own and run it in a portable way.

Configure and build

After fetching sources, I tried configuring and building with CMake (under VSCode):
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DTHEROCK_ENABLE_RCCL=OFF -DTHEROCK_ENABLE_MATH_LIBS=OFF -DTHEROCK_ENABLE_ML_FRAMEWORKS=OFF -DTHEROCK_AMDGPU_FAMILIES=gfx110X-dgpu -UTHEROCK_AMDGPU_TARGETS -UTHEROCK_AMDGPU_DIST_BUNDLE_NAME -DLLVM_BUILD_LLVM_DYLIB=OFF --no-warn-unused-cli -SD:/projects/TheRock -Bd:/projects/TheRock/build -G Ninja

I found a few Windows issues specific to TheRock:

• Quote paths in toolchain to support spaces in paths on Windows. #34
• Skip dlopen test on Windows for now. #35

Next, I'm finding issues in the subprojects that TheRock includes. Sample logs: https://gist.github.com/ScottTodd/0a6625c4502169c5d54535bf122fc7fd. I'm not sure how deep these issues go, and if some of them are fixed on development branches. Here are a few I've found so far:

• rocm-core __attribute__ is not portable to all compilers (specifically MSVC) in https://github.com/ROCm/rocm-core/blob/master/rocm_version.h.in, do something like

  #if defined(_WIN32) || defined(__CYGWIN__)
  #define LIB_API_PUBLIC __declspec(dllexport)
  #else
  #define LIB_API_PUBLIC __attribute__((visibility("default")))
  #endif

  Need a similar fix for __attribute__((nonnull)), maybe _Ret_nonnull (https://stackoverflow.com/a/78737973)

• rocm-core link.h and dlfcn.h headers are Unix-only in https://github.com/ROCm/rocm-core/blob/master/rocm_getpath.cpp (use something like LoadLibraryA() on a .dll instead of dlopen() on a .so). The headers should also only be included if BUILD_SHARED_LIBS is defined (see next bullet)

  • Setting -DBUILD_SHARED_LIBS=OFF here helps a bit: https://github.com/nod-ai/TheRock/blob/96ebf46b5ca62add1307d9f03db6d04f31207ff0/base/CMakeLists.txt#L22

• rocm-core in the same file uses PATH_MAX which is nonstandard. There is a FILENAME_MAX that is standard but https://stackoverflow.com/a/65174437 says it is not to be trusted for memory allocation. Could do something like min(FILENAME_MAX, 4096) (choosing some sensible value that is not INT_MAX)

• llvm warns about Generating libLLVM is not supported on MSVC in https://github.com/ROCm/llvm-project/blob/amd-staging/llvm/tools/llvm-shlib/CMakeLists.txt so I set -DLLVM_BUILD_LLVM_DYLIB=OFF in my CMake configure command. Not sure if that was load bearing for anything else yet. Could auto-detect and flip that off by default in TheRock or one of the subprojects that includes it.

• rocm_smi_lib sets CMAKE_CXX_FLAGS that assume gcc/clang: https://github.com/ROCm/rocm_smi_lib/blob/e5c5a1d5b7cba41ab801a987fb6c466503411804/CMakeLists.txt#L79-L85. It seems like https://github.com/ROCm/amdsmi is the successor to rocm_smi, but it has the same issue: https://github.com/ROCm/amdsmi/blob/9872083b491c9b136496fd493414aca5c6c144cb/CMakeLists.txt#L99-L104. Both projects also only claim to support Linux.

[ScottTodd]

Got a build to succeed (run to completion with exit code 0) after disabling many subprojects and writing some local patches.

• Logs: https://gist.github.com/ScottTodd/3ad6ea3caae56c146856bfb8225f1844
• Branch with changes: https://github.com/ScottTodd/TheRock/tree/windows-bootstrap
• Still need to figure out how to export my patches into the https://github.com/nod-ai/TheRock/tree/main/patches to share them. Mostly working around the rocm-core issues described above

[amd-chrissosa]

For 3: git diff master Branch1 > ../patchfile and check that in should be fine. apply patches functionality seems pretty straightforward: https://github.com/nod-ai/TheRock/blob/main/build_tools/fetch_sources.py

[stellaraccident]

FYI - for managing the patches directory, I've been making commits to the repo checked out git dirs. Then you dump them all with git format-patch. There's a handy (Linux) script in build_tools/save_patches.sh, but it can just be a one liner:

git format-patch -o ~/TheRock/patches/rocm-6.3.1/rocm-core rocm-6.3.1

Then if you re-run fetch_sources.py it will reset to the base tag (rocm-6.3.1) and re-apply the patches, same as a fresh checkout on the CI.

I was doing it the manual way via git diff but it was absolutely murder trying to upstream them and such. This way, you have a nice stack of commits in your git repo and can cherry-pick/rebase to a new branch to make an upstream PR, etc.

[stellaraccident]

Sorry - I could have documented my flow on this better: key is that git format-patch is the inverse of git am (which is what fetch_sources.py does to re-apply)

[ScottTodd]

Thanks, the save_patches.sh script worked fine for me (using git bash under cmder)

[ScottTodd]

Made some progress on pushing patches needed for a minimal local build. Now trying to get that wrapped into a GitHub Actions workflow, using this dev branch: https://github.com/ScottTodd/TheRock/tree/windows-ci-setup (workflow file is here: https://github.com/ScottTodd/TheRock/blob/windows-ci-setup/.github/workflows/build_windows_packages.yml). Here are some logs with a bunch of debug steps included: https://github.com/ScottTodd/TheRock/actions/runs/13082919002/job/36509861561.

Stuck for today on

Run cmake --build build
[0/2] Re-checking globbed directories...
ninja: error: 'D:/a/TheRock/TheRock/base/rocm-cmake/CMakeLists.txt', needed by 'compile_commands_fragment_rocm-cmake.json', missing and no known rule to make it
Error: Process completed with exit code 1.

I saw that locally when symlinks were not enabled in the git repository or the fetch_sources.py file was not run. I think the right files are on the runner, but I'm still suspicious of the symlink / hardlink configuration.

[stellaraccident]

Made some progress on pushing patches needed for a minimal local build. Now trying to get that wrapped into a GitHub Actions workflow, using this dev branch: https://github.com/ScottTodd/TheRock/tree/windows-ci-setup (workflow file is here: https://github.com/ScottTodd/TheRock/blob/windows-ci-setup/.github/workflows/build_windows_packages.yml). Here are some logs with a bunch of debug steps included: https://github.com/ScottTodd/TheRock/actions/runs/13082919002/job/36509861561.

Stuck for today on

Run cmake --build build
[0/2] Re-checking globbed directories...
ninja: error: 'D:/a/TheRock/TheRock/base/rocm-cmake/CMakeLists.txt', needed by 'compile_commands_fragment_rocm-cmake.json', missing and no known rule to make it
Error: Process completed with exit code 1.

I saw that locally when symlinks were not enabled in the git repository or the fetch_sources.py file was not run. I think the right files are on the runner, but I'm still suspicious of the symlink / hardlink configuration.

That is a strange error: there should not be a direct dependency between those two files (the dependency crosses a couple of other commands). But I think this is cmake/ninja wonkiness where for custom targets, it makes them depend on the transitive set of input files. So there is probably a lot wrong, and you're just getting that one error out of the pile randomly.

One thing you could quickly try: on line 418 of therock_subproject.cmake, comment out the `"${_cmake_source_dir}/CMakeLists.txt" line. This will mean it won't retrigger on sub-project changes, but that is not a problem on CI.

Still feeling like there is something else wrong and this is just the error you are getting.

I can pull it down on my windows machine and try.