Sigstore warnings on stderr errored the release process

[hugovk]

hugovk@downloads:~$ python3 --version --version
Python 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0]
hugovk@downloads:~$ python3 -m sigstore --version
sigstore 3.4.0

This happened during the 3.14.0a1 release:

✅  Push new tags and branches to upstream
✅  Removing temporary release branch
Waiting for browser interaction...
💥  Add files to python.org download page
Traceback (most recent call last):
  File "/Users/hugo/github/release-tools/run_release.py", line 1340, in <module>
    main()
  File "/Users/hugo/github/release-tools/run_release.py", line 1336, in main
    automata.run()
  File "/Users/hugo/github/release-tools/run_release.py", line 245, in run
    raise e from None
  File "/Users/hugo/github/release-tools/run_release.py", line 242, in run
    self.current_task(self.db)
  File "/Users/hugo/github/release-tools/release.py", line 126, in __call__
    return getattr(self, "function")(db)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hugo/github/release-tools/run_release.py", line 966, in run_add_to_python_dot_org
    raise paramiko.SSHException(f"Failed to execute the command: {stderr_text}")
paramiko.ssh_exception.SSHException: Failed to execute the command: OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-embed-win32.zip
[23:13:23] WARNING  /srv/www.python.org/ftp/python/3.14.0/python-3.1 _cli.py:925
                    4.0a1-embed-win32.zip:
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-embed-win32.zip.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-embed-win32.zip.sigstore.json. Support for
                    discovering 'bare' .sigstore inputs will be
                    deprecated in a future release.
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-embed-win32.zip
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-embed-amd64.zip
[23:13:26] WARNING  /srv/www.python.org/ftp/python/3.14.0/python-3.1 _cli.py:925
                    4.0a1-embed-amd64.zip:
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-embed-amd64.zip.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-embed-amd64.zip.sigstore.json. Support for
                    discovering 'bare' .sigstore inputs will be
                    deprecated in a future release.
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-embed-amd64.zip
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a1.tgz
[23:13:29] WARNING  /srv/www.python.org/ftp/python/3.14.0/Python-3.1 _cli.py:925
                    4.0a1.tgz:
                    /srv/www.python.org/ftp/python/3.14.0/Python-3.1
                    4.0a1.tgz.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/Python-3.1
                    4.0a1.tgz.sigstore.json. Support for discovering
                    'bare' .sigstore inputs will be deprecated in a
                    future release.
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a1.tgz
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-arm64.exe
[23:13:31] WARNING  /srv/www.python.org/ftp/python/3.14.0/python-3.1 _cli.py:925
                    4.0a1-arm64.exe:
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-arm64.exe.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-arm64.exe.sigstore.json. Support for
                    discovering 'bare' .sigstore inputs will be
                    deprecated in a future release.
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-arm64.exe
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-macos11.pkg
[23:13:34] WARNING  /srv/www.python.org/ftp/python/3.14.0/python-3.1 _cli.py:925
                    4.0a1-macos11.pkg:
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-macos11.pkg.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-macos11.pkg.sigstore.json. Support for
                    discovering 'bare' .sigstore inputs will be
                    deprecated in a future release.
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-macos11.pkg
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1.exe
[23:13:36] WARNING  /srv/www.python.org/ftp/python/3.14.0/python-3.1 _cli.py:925
                    4.0a1.exe:
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1.exe.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1.exe.sigstore.json. Support for discovering
                    'bare' .sigstore inputs will be deprecated in a
                    future release.
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1.exe
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a1.tar.xz
[23:13:38] WARNING  /srv/www.python.org/ftp/python/3.14.0/Python-3.1 _cli.py:925
                    4.0a1.tar.xz:
                    /srv/www.python.org/ftp/python/3.14.0/Python-3.1
                    4.0a1.tar.xz.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/Python-3.1
                    4.0a1.tar.xz.sigstore.json. Support for
                    discovering 'bare' .sigstore inputs will be
                    deprecated in a future release.
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a1.tar.xz
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-amd64.exe
[23:13:41] WARNING  /srv/www.python.org/ftp/python/3.14.0/python-3.1 _cli.py:925
                    4.0a1-amd64.exe:
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-amd64.exe.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-amd64.exe.sigstore.json. Support for
                    discovering 'bare' .sigstore inputs will be
                    deprecated in a future release.
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-amd64.exe
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-embed-arm64.zip
[23:13:43] WARNING  /srv/www.python.org/ftp/python/3.14.0/python-3.1 _cli.py:925
                    4.0a1-embed-arm64.zip:
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-embed-arm64.zip.sigstore should be named
                    /srv/www.python.org/ftp/python/3.14.0/python-3.1
                    4.0a1-embed-arm64.zip.sigstore.json. Support for
                    discovering 'bare' .sigstore inputs will be
                    deprecated in a future release.
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a1-embed-arm64.zip

It's from here:

release-tools/run_release.py

Lines 960 to 965 in 2eea7cf

	stdin, stdout, stderr = client.exec_command(
	f"AUTH_INFO={auth_info} SIGSTORE_IDENTITY_TOKEN={identity_token} python3 add_to_pydotorg.py {db['release']}"
	)
	stderr_text = stderr.read().decode()
	if stderr_text:
	raise paramiko.SSHException(f"Failed to execute the command: {stderr_text}")

It wasn't really an error, but there was warning output on stderr, which the script interpreted as a terminal error.

I temporarily worked around it by commenting the guard out:

    stderr_text = stderr.read().decode()
    # if stderr_text:
    #     raise paramiko.SSHException(f"Failed to execute the command: {stderr_text}")

[sethmlarson]

So this warning is a result of: sigstore/sigstore-python#1178

I'm not sure there's much we can do besides suppressing the stderr-to-error behavior for that specific command for now.

[hugovk]

So this warning is a result of: sigstore/sigstore-python#1178

Fixed in sigstore/sigstore-python#1179 and released in https://github.com/sigstore/sigstore-python/releases/tag/v3.5.0.

Should we update the scripts to require 3.5.0 on the downloads server?

For example, around:

release-tools/add_to_pydotorg.py

Lines 352 to 357 in 00dd4d7

	if sigstore_major_version < 3:
	error(
	f"Sigstore v3 or later must be installed "
	f"(currently {sigstore_version}), "
	f"run: python -m pip install -r requirements.txt"
	)

This could also tie into #179 around auto-installing or failing early etc.

[hugovk]

I forgot about this, so just manually upgraded to sigstore-3.5.1 on the server for 3.14.0a2.

Then I got this:

✅  Removing temporary release branch
Waiting for browser interaction...
💥  Add files to python.org download page
Traceback (most recent call last):
  File "/Users/hugo/github/release-tools/run_release.py", line 1317, in <module>
    main()
  File "/Users/hugo/github/release-tools/run_release.py", line 1313, in main
    automata.run()
  File "/Users/hugo/github/release-tools/run_release.py", line 245, in run
    raise e from None
  File "/Users/hugo/github/release-tools/run_release.py", line 242, in run
    self.current_task(self.db)
  File "/Users/hugo/github/release-tools/release.py", line 126, in __call__
    return getattr(self, "function")(db)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hugo/github/release-tools/run_release.py", line 945, in run_add_to_python_dot_org
    raise paramiko.SSHException(f"Failed to execute the command: {stderr_text}")
paramiko.ssh_exception.SSHException: Failed to execute the command: OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2.exe
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2.exe
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a2.tgz
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a2.tgz
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a2.tar.xz
OK: /srv/www.python.org/ftp/python/3.14.0/Python-3.14.0a2.tar.xz
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-amd64.exe
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-amd64.exe
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-embed-amd64.zip
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-embed-amd64.zip
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-macos11.pkg
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-macos11.pkg
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-embed-arm64.zip
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-embed-arm64.zip
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-arm64.exe
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-arm64.exe
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-embed-win32.zip
OK: /srv/www.python.org/ftp/python/3.14.0/python-3.14.0a2-embed-win32.zip

That's not an error, but the script again assumes any output from the command is an error and fails.

Like last time, I temporarily worked around it by commenting the guard out:

    stderr_text = stderr.read().decode()
    # if stderr_text:
    #     raise paramiko.SSHException(f"Failed to execute the command: {stderr_text}")