Anaconda Cloud now causes error and package removal for upload of package version of same name

[matthewfeickert]

Copying scientific-python/upload-nightly-action#57 here to get feedback from the Anaconda team. This is a breaking behavior change, so advice on how to proceed would be quite welcome.

Anaconda Cloud seems to have changed how they handle uploads of packages where there is already a package of the same name and version on Anaconda Cloud. In the past, this simply just resulted in the artifact being uploaded overwriting the previous artifact. networkx is one of the core packages that used this approach. However, I noticed that networkx's nightly workflow is failing to do so

...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "networkx"
Creating release "3.3rc0.dev0"
Uploading file "scientific-python-nightly-wheels/networkx/3.3rc0.dev0/networkx-3.3rc0.dev0-py3-none-any.whl"
Warning:  Distribution "networkx-3.3rc0.dev0-py3-none-any.whl" already exists. Removing.
Error:  ("release version='3.3rc0.dev0' does not exist", 404)
##[debug]Docker Action run completed with exit code 1
##[debug]Finishing: Upload nighlty wheel

(The success after failure is due to the package being removed and so there is no conflict the next upload.)

This started happening on 2024-01-04 for networkx, and was not happening before that. For example, this 2024-01-02 upload ran successfully

+ anaconda --token *** upload --force --user scientific-python-nightly-wheels dist/networkx-3.3rc0.dev0-py3-none-any.whl
...
Using Anaconda API: https://api.anaconda.org/
Using "scientific-python-nightly-wheels" as upload username
Processing "dist/networkx-3.3rc0.dev0-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "networkx"
Creating release "3.3rc0.dev0"
Uploading file "scientific-python-nightly-wheels/networkx/3.3rc0.dev0/networkx-3.3rc0.dev0-py3-none-any.whl"
Warning:  Distribution "networkx-3.3rc0.dev0-py3-none-any.whl" already exists. Removing.

  0%|          | 0.00/1.59M [00:00<?, ?B/s]
1.59MB [00:00, 5.26MB/s]                   
Upload complete

standard python located at:
  https://anaconda.org/scientific-python-nightly-wheels/networkx

We know that as we're using a lock file that nothing has changed from the upload-nightly-action action side, and so this has to be on the Anaconda Cloud side.

[matthewfeickert]

This behavior also seems to be invalidating the --force flag of anaconda upload

$ anaconda --version
anaconda Command line client (version 1.12.1)
$ anaconda upload --help
usage: anaconda upload [-h] [-c CHANNELS] [-l LABELS] [--no-progress] [-u USER] [--keep-basename] [-p PACKAGE] [-v VERSION] [-s SUMMARY] [-t PACKAGE_TYPE] [-d DESCRIPTION] [--thumbnail THUMBNAIL]
                       [--private] [--no-register | --register] [--build-id BUILD_ID] [-i | -f | --force | --skip-existing | -m]
                       files [files ...]

Upload packages to your Anaconda repository

positional arguments:
  files                 Distributions to upload

options:
  -h, --help            show this help message and exit
  -c CHANNELS, --channel CHANNELS
                        [DEPRECATED] Add this file to a specific channel. Warning: if the file channels do not include "main", the file will not show up in your user channel
  -l LABELS, --label LABELS
                        Add this file to a specific label. Warning: if the file labels do not include "main", the file will not show up in your user label
  --no-progress         Don't show upload progress
  -u USER, --user USER  User account or Organization, defaults to the current user
  --keep-basename       Do not normalize a basename when uploading a conda package.
  --no-register         Don't create a new package namespace if it does not exist
  --register            Create a new package namespace if it does not exist
  --build-id BUILD_ID   Anaconda repository Build ID (internal only)
  -i, --interactive     Run an interactive prompt if any packages are missing
  -f, --fail            Fail if a package or release does not exist (default)
  --force               Force a package upload regardless of errors
  --skip-existing       Skip errors on package batch upload if it already exists
  -m, --force-metadata-update
                        Overwrite existing release metadata with the metadata from the package.
...

which is being used in the relevant code of scientific-python/upload-nightly-action#57.

[matthewfeickert]

@csoja @yshmatov-anaconda @vshevchenko-anaconda Can you please comment here, or tag people on the relevant teams? This is API breaking behavior changes and so quite serious.

[matthewfeickert]

@kenodegard @dholth as you're currently reviewing PRs for the project, can you please either comment on this or notify the relevant teams?

[dholth]

We have notified the relevant teams.

[matthewfeickert]

We have notified the relevant teams.

Thanks very much @dholth. I appreciate it.

[matthewfeickert]

What is intersting about scikit-hep/awkward#3012 (comment) though, where you are attempting to upload mutliple wheels that have the same name each run of the uploader (no vcs information in the wheel name), is that it seems that for repeated uploads things aren't fully replicable. In the case of Awkward uploading to the scikit-hep-nightly-wheels Anaconda cloud org, running the workflow in CI on a schedule results in failures that match what this Issue is describing:

first run of workflow

+ anaconda --token *** upload --force --user scikit-hep-nightly-wheels --label main dist/awkward-2.6.1-py3-none-any.whl dist/awkward_cpp-29-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl dist/awkward_cpp-29-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl dist/awkward_cpp-29-cp38-cp38-macosx_10_9_universal2.whl dist/awkward_cpp-29-cp39-cp39-win_amd64.whl
Uploading wheels to anaconda.org...
Using Anaconda API: https://api.anaconda.org/
Using "scikit-hep-nightly-wheels" as upload username
Processing "dist/awkward-2.6.1-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "awkward"
Creating release "2.6.1"
Uploading file "scikit-hep-nightly-wheels/awkward/2.6.1/awkward-2.6.1-py3-none-any.whl"
Warning:  Distribution "awkward-2.6.1-py3-none-any.whl" already exists. Removing.
Error:  ("release version='2.6.1' does not exist", 404)

However, if I then trigger a rerun of the workflow through GitHub Actions, the upload of awkward-2.6.1-py3-none-any.whl this time works, for the reasons discussed in scientific-python/upload-nightly-action#57 (comment), but I would suspect that all of the awkward-cpp wheels should now fail as they haven't yet tried, and failed, to be uploaded as the workflow errored out immediatley. However, they seem to succeed (as one would expect from use of --force), even though they have this same

Warning:  Distribution "X" already exists. Removing.

second run of workflow

+ anaconda --token *** upload --force --user scikit-hep-nightly-wheels --label main dist/awkward-2.6.1-py3-none-any.whl dist/awkward_cpp-29-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl dist/awkward_cpp-29-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl dist/awkward_cpp-29-cp38-cp38-macosx_10_9_universal2.whl dist/awkward_cpp-29-cp39-cp39-win_amd64.whl
Uploading wheels to anaconda.org...
Using Anaconda API: https://api.anaconda.org/
Using "scikit-hep-nightly-wheels" as upload username
Processing "dist/awkward-2.6.1-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "awkward"
Creating release "2.6.1"
Uploading file "scikit-hep-nightly-wheels/awkward/2.6.1/awkward-2.6.1-py3-none-any.whl"

  0%|          | 0.00/744k [00:00<?, ?B/s]
  2%|▏         | 16.0k/744k [00:00<00:11, 67.3kB/s]
 15%|█▌        | 112k/744k [00:00<00:01, 350kB/s]  
 43%|████▎     | 320k/744k [00:00<00:00, 799kB/s]
750kB [00:00, 964kB/s]                           
Upload complete

Processing "dist/awkward_cpp-29-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "awkward-cpp"
Creating release "29"
Uploading file "scikit-hep-nightly-wheels/awkward-cpp/29/awkward_cpp-29-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
Warning:  Distribution "awkward_cpp-29-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" already exists. Removing.

  0%|          | 0.00/690k [00:00<?, ?B/s]
  2%|▏         | 16.0k/690k [00:00<00:09, 71.7kB/s]
 16%|█▌        | 112k/690k [00:00<00:01, 364kB/s]  
 46%|████▋     | 320k/690k [00:00<00:00, 822kB/s]
695kB [00:00, 1.02MB/s]                          
Upload complete

Processing "dist/awkward_cpp-29-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "awkward-cpp"
Creating release "29"
Uploading file "scikit-hep-nightly-wheels/awkward-cpp/29/awkward_cpp-29-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
Warning:  Distribution "awkward_cpp-29-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" already exists. Removing.

  0%|          | 0.00/691k [00:00<?, ?B/s]
  2%|▏         | 16.0k/691k [00:00<00:10, 65.8kB/s]
 16%|█▌        | 112k/691k [00:00<00:01, 341kB/s]  
 49%|████▊     | 336k/691k [00:00<00:00, 835kB/s]
697kB [00:00, 849kB/s]                           
Upload complete

Processing "dist/awkward_cpp-29-cp38-cp38-macosx_10_9_universal2.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "awkward-cpp"
Creating release "29"
Uploading file "scikit-hep-nightly-wheels/awkward-cpp/29/awkward_cpp-29-cp38-cp38-macosx_10_9_universal2.whl"
Warning:  Distribution "awkward_cpp-29-cp38-cp38-macosx_10_9_universal2.whl" already exists. Removing.

  0%|          | 0.00/1.16M [00:00<?, ?B/s]
  1%|▏         | 16.0k/1.16M [00:00<00:17, 67.4kB/s]
  9%|▉         | 112k/1.16M [00:00<00:03, 351kB/s]  
 30%|██▉       | 352k/1.16M [00:00<00:00, 891kB/s]
 51%|█████▏    | 608k/1.16M [00:00<00:00, 1.31MB/s]
 84%|████████▎ | 992k/1.16M [00:00<00:00, 1.82MB/s]
1.16MB [00:00, 1.25MB/s]                           
Upload complete

Processing "dist/awkward_cpp-29-cp39-cp39-win_amd64.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "awkward-cpp"
Creating release "29"
Uploading file "scikit-hep-nightly-wheels/awkward-cpp/29/awkward_cpp-29-cp39-cp39-win_amd64.whl"
Warning:  Distribution "awkward_cpp-29-cp39-cp39-win_amd64.whl" already exists. Removing.

  0%|          | 0.00/521k [00:00<?, ?B/s]
  3%|▎         | 16.0k/521k [00:00<00:08, 59.6kB/s]
 21%|██▏       | 112k/521k [00:00<00:01, 327kB/s]  
526kB [00:00, 1.33MB/s]                          
526kB [00:00, 711kB/s] 
Upload complete

standard python located at:
  https://anaconda.org/scikit-hep-nightly-wheels/awkward

standard python located at:
  https://anaconda.org/scikit-hep-nightly-wheels/awkward-cpp

standard python located at:
  https://anaconda.org/scikit-hep-nightly-wheels/awkward-cpp

standard python located at:
  https://anaconda.org/scikit-hep-nightly-wheels/awkward-cpp

standard python located at:
  https://anaconda.org/scikit-hep-nightly-wheels/awkward-cpp

This is quite strange.

(cc @jpivarski)

[matthewfeickert]

@dholth Ah. I think I understand what is happening now (summarized in scientific-python/upload-nightly-action#57 (comment)). This might already have been obvious to you/the relevant teams.

The package directory is getting removed by the backend in mid-upload as it detects that the only file in the package has been removed. This somehow changed in January 2024(?). Is it possible to have this cleanup logic know that it got triggered during an upload attempt and have it wait until after the upload finishes to see if it should delete the package from the registry?

In the anaconda-client code this is just

https://github.com/Anaconda-Platform/anaconda-client/blob/d9849e3a59c575518bad0001d97556aec9512803/binstar_client/commands/upload.py#L590-L592

https://github.com/Anaconda-Platform/anaconda-client/blob/d9849e3a59c575518bad0001d97556aec9512803/binstar_client/__init__.py#L475-L484

where self.session is just a simple requests session

https://github.com/Anaconda-Platform/anaconda-client/blob/d9849e3a59c575518bad0001d97556aec9512803/binstar_client/__init__.py#L42-L46

to interact with https://api.anaconda.org (where I guess the API behavior change that we're seeing took place?).

[matthewfeickert]

Closed as release v1.12.3 fixes this (specifically PR #708). 👍