Use GitHub Actions artifacts in Anaconda uploads #364

jameslamb · 2025-05-22T19:18:46Z

Contributes to rapidsai/build-planning#181

Replaces #332

There are 2 workflows left here that depend on conda packages and wheels being uploaded to downloads.rapids.ai:

conda-upload-packages: uploads conda packages to the rapidsai / rapidsai-nightly channels
wheels-publish: uploads wheels to https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/ (and sometimes pypi.org)

This PR updates those workflows to download the packages from GitHub Actions artifacts instead of S3.

Notes for Reviewers

This is intentionally targeting `branch-25.08`

I'm opening this on the first day of code freeze for the 25.06 release.
To avoid blocking that release, I'm intentionally targeting this change at 25.08.

Deployment Plan

merge stop uploading packages to downloads.rapids.ai cugraph-gnn#215
check that the conda and wheel publishing on the corresponding branch build works
check that the conda and wheel publishing on the next nightly build works
merge this PR (do not delete its branch though)
push a follow-up PR in cugraph-gnn reverting references to that gha-artifacts/uploads branch of shared-workflows
go remote all the remaining uses or rapids-upload-wheels-to-s3 and rapids-upload-conda-to-s3 across RAPIDS (also on branch-25.08).

Why use `cugraph-gnn` to test this?

Proposing that cugraph-gnn be used to test this pattern for all of RAPIDS.

It's a good candidate to test this for the following reasons:

very few downstream dependencies (so its nightly publishing being broken would be less disruptive than, say, rmm's)
has C++ and Python wheels and conda packages
nightly CI is currently succeeding on branch-25.08

How I tested this

See "Deployment Plan"... I'd like to merge rapidsai/cugraph-gnn#215 to test this.

Looks to me like this worked on cugraph-gnn: rapidsai/cugraph-gnn#215 (comment)

Contributes to rapidsai/build-planning#181 Proposes stopping all uploads of wheels and conda packages to `downloads.rapids.ai`, in favor of using the GitHub Actions artifact store. ## Notes for Reviewers ### Intentionally targeting `branch-25.08` To avoid disrupting the 25.06 release. ### How I found the lines to change ```shell git grep -E '\-s3' git grep 'upload' git grep -E 'downloads\.rapids' ``` ### Why `cugraph-gnn`? See "Deployment Plan" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #215

jameslamb · 2025-05-27T16:37:34Z

It looks like this is working: rapidsai/cugraph-gnn#215 (comment)

This could be merged at any time. Then I'll put up follow-up PRs across RAPIDS removing uploads to downloads.rapids.ai.

jameslamb · 2025-05-27T17:35:04Z

.github/workflows/conda-upload-packages.yaml

-        with:
-          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
-          aws-region: ${{ vars.AWS_REGION }}
-          role-duration-seconds: 43200 # 12h


I just realized, we probably could remove the step getting AWS creds from these upload workflows as well! Now that they shouldn't need to interact directly with any AWS services.

Wish I'd thought of that before merging rapidsai/cugraph-gnn#215 ... I'll kick off another build there to test that theory.

Triggered a new nightly run of cugraph-gnn like this:

Using this commit: rapidsai/cugraph-gnn@915497b

Build link: https://github.com/rapidsai/cugraph-gnn/actions/runs/15281964715/job/42982961992

Yep looked to me like that all worked without AWS creds 😁

https://github.com/rapidsai/cugraph-gnn/actions/runs/15281964715/job/42983516974

Contributes to rapidsai/build-planning#181 Follow-up to #215 Now that rapidsai/shared-workflows#364 has been merged to `branch-25.08`, this project's CI configurations should stop pointing at the branch from that PR. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #216

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #883

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #6803

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Gil Forsyth (https://github.com/gforsyth) - Bradley Dice (https://github.com/bdice) URL: #940

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1501

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #734

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #145

## Description Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` * updates to `branch-25.08` of `shared-workflows`, to pull in rapidsai/shared-workflows#364 ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) URL: #112

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #308

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` Also updates a few RAPIDS `pre-commit` hook versions, while were a bit out of date. ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #692

…lows (#144) Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` * updates all `shared-workflows` references to `branch-25.08`, to pull in changes from rapidsai/shared-workflows#364 ## Notes for Reviewers ### Is it safe to update to `branch-25.08` of `shared-workflows` on `main` here? I'm assuming yes, because CI here computes its own custom-to-`pynvjitlink` support matrices: https://github.com/rapidsai/pynvjitlink/blob/8ea3d166fa2c0cc9372829a84c9e5d5721988895/.github/workflows/build.yaml#L16-L24 All other non-matrix changes on `shared-workflows` between `branch-25.06` and `branch-25.08` should be fine for this repo. ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #144

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #2688

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1141

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #428

## Description Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #18973

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#18973

#50) Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` (these are not used by anything in this repo any more) Contributes to rapidsai/build-planning#178 * prevents triggering expensive CI jobs based on PRs modifying `ci/release/update-version-*.sh` Contributes to rapidsai/build-planning#179 * adds a call to `rapids-init-pip` near the beginning of all CI scripts that use `pip` * removes redefinitions of `PIP_CONSTRAINT` in scripts, in favor of using the one set up by `rapids-pip-init` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 ### Haven't I seen these changes before? Yes, some of these changes were made in `rapidsai/cuopt` (rapidsai/cuopt#2503) but looks like they were lost when this new repo was created. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Ramakrishnap (https://github.com/rgsl888prabhu) URL: #50

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` Contributes to rapidsai/build-planning#135 * adds `shellcheck` checks to `pre-commit` configuration ~Also proposes changes to get around the cold-start problem we have at the beginning of every release, where `test-conda-nightly-env` cannot run because there aren't yet published `rapids` packages.~ * ~makes `test-conda-nightly-env` dependent on the just-built-in-CI `rapids` packages, instead of installing them from `rapidsai-nightly`~ ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #770

Contributes to rapidsai/build-planning#181 * removes all uploads of conda packages and wheels to `downloads.rapids.ai` ## Notes for Reviewers ### How I identified changes Looked for uses of the relevant `gha-tools` tools, as well as documentation about `downloads.rapids.ai`, being on the NVIDIA VPN, using S3, etc. like this: ```shell git grep -i -E 's3|upload|downloads\.rapids|vpn' ``` ### How I tested this See "How I tested this" on rapidsai/shared-workflows#364 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Gil Forsyth (https://github.com/gforsyth) - Bradley Dice (https://github.com/bdice) URL: rapidsai#940

Use GitHub Actions artifacts in Anaconda uploads

2d5c655

jameslamb added breaking Introduces a breaking change improvement Improves an existing functionality labels May 22, 2025

This was referenced May 22, 2025

Publish packages from Github Artifact Store #332

Closed

Stop uploading wheels and conda packages to downloads.rapids.ai rapidsai/build-planning#181

Closed

stop uploading packages to downloads.rapids.ai rapidsai/cugraph-gnn#215

Merged

jameslamb changed the title ~~WIP: Use GitHub Actions artifacts in Anaconda uploads~~ [DO NOT MERGE] Use GitHub Actions artifacts in Anaconda uploads May 27, 2025

Merge branch 'branch-25.08' into gha-artifacts/uploads

e1ea652

jameslamb changed the title ~~[DO NOT MERGE] Use GitHub Actions artifacts in Anaconda uploads~~ Use GitHub Actions artifacts in Anaconda uploads May 27, 2025

jameslamb requested a review from ajschmidt8 May 27, 2025 16:37

jameslamb marked this pull request as ready for review May 27, 2025 16:37

jameslamb requested a review from a team as a code owner May 27, 2025 16:37

remove AWS auth steps

14552cf

jameslamb commented May 27, 2025

View reviewed changes

This was referenced May 27, 2025

stop uploading packages to downloads.rapids.ai rapidsai/rmm#1929

Merged

stop uploading packages to downloads.rapids.ai rapidsai/nx-cugraph#145

Merged

stop uploading packages to downloads.rapids.ai rapidsai/cuvs#940

Merged

ajschmidt8 approved these changes May 27, 2025

View reviewed changes

jameslamb merged commit e62a069 into branch-25.08 May 27, 2025
2 checks passed

This was referenced May 27, 2025

use branch-25.08 of shared-workflows rapidsai/cugraph-gnn#216

Merged

stop uploading packages to downloads.rapids.ai rapidsai/cucim#883

Merged

stop uploading to downloads.rapids.ai, add shellcheck rapidsai/rapids-metadata#54

Merged

This was referenced May 27, 2025

stop uploading packages to downloads.rapids.ai rapidsai/jupyterlab-nvdashboard#236

Merged

stop uploading packages to downloads.rapids.ai, update to 25.08 workflows rapidsai/rapids-cli#98

Merged

jameslamb mentioned this pull request May 27, 2025

stop uploading packages to downloads.rapids.ai rapidsai/raft#2688

Merged

jameslamb deleted the gha-artifacts/uploads branch May 28, 2025 01:51

jameslamb mentioned this pull request May 29, 2025

add wheel publishing rapidsai/rapidsmpf#319

Merged

jameslamb mentioned this pull request May 30, 2025

stop uploading to downloads.rapids.ai, use rapids-init-pip in wheel CI NVIDIA/cuopt#50

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use GitHub Actions artifacts in Anaconda uploads #364

Use GitHub Actions artifacts in Anaconda uploads #364

Uh oh!

jameslamb commented May 22, 2025 •

edited

Loading

Uh oh!

jameslamb commented May 27, 2025

Uh oh!

jameslamb May 27, 2025

Uh oh!

jameslamb May 27, 2025

Uh oh!

jameslamb May 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Use GitHub Actions artifacts in Anaconda uploads #364

Use GitHub Actions artifacts in Anaconda uploads #364

Uh oh!

Conversation

jameslamb commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes for Reviewers

This is intentionally targeting branch-25.08

Deployment Plan

Why use cugraph-gnn to test this?

How I tested this

Uh oh!

jameslamb commented May 27, 2025

Uh oh!

jameslamb May 27, 2025

Choose a reason for hiding this comment

Uh oh!

jameslamb May 27, 2025

Choose a reason for hiding this comment

Uh oh!

jameslamb May 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jameslamb commented May 22, 2025 •

edited

Loading

This is intentionally targeting `branch-25.08`

Why use `cugraph-gnn` to test this?