Auto updater wants to update to a pre-release version

[steven166]

Describe the bug
The auto updater of the gha runner want to upgrade to a prerelease

To Reproduce

1. Start the runner on version 2.263.0
2. It will show the following message in the output:

Runner update in progress, do not shutdown runner.
Downloading 2.267.1 runner

Expected behavior
I don't want auto update at all, but being auto updated to a pre-release version is far from acceptable.
At least I only want to be updated to a stable version.

Runner Version and Platform

2.263.0

OS of the machine running the runner? OSX/Windows/Linux/...
Linux

[j3parker]

We're fine with auto-updating, but when it happens it can mean extra latency before accepting builds since we auto-scale our runners and plan to terminate them after each build (the --once option). We'd like to know which exe we should be baking into our VMs (with Packer) so that we can typically avoid this.

We have the runner version pinned in our Packer build. We poll https://api.github.com/repos/actions/runner/releases/latest hourly. When there is a new version we update our build. We use virtual-environments to configure our VMs and all-told it takes about 4 hours to build (for Linux). So in total it can take us up to 5 hours before we have the latest runner on our VMs.

Should we be using the pre-release versions instead? We don't care either way.

In a perfect world we'd get ~5 hours to update our VMs on our own before an update was forced but it's not the end of the world for us if they update automatically sooner (e.g. when you have urgent patches). We'd just like clarity on which channel to track for our VMs.

[npalm]

We face the same issue, auto upgrade is ok but auto upgrade to a pre release is not OK to us. We auto scal up and down runners and cache the distribution for perfornamce reasons. This cache is sync with releases, and not with pre releases. Please can you explain why the 263 runner version is auto updating to a pre release?

[erulabs]

Auto-upgrade is a bit of a nightmare - causes github-runner to be fairly unreliable when running inside a container. I was previously using a hack where I would upgrade to the latest runner whenever my runner boots up, thus avoiding the default crash-loop behavior (update -> crash -> start with old version -> repeat). The entrypoint for my runner looks like:

#!/bin/sh

# Try to download latest update on startup
GITHUB_RUNNER_VERSION=$(curl -s https://api.github.com/repos/actions/runner/releases/latest | jq '.tag_name' --raw-output | cut -c2-)
echo "Downloading Github Runner ${GITHUB_RUNNER_VERSION}"
curl -Ls https://github.com/actions/runner/releases/download/v${GITHUB_RUNNER_VERSION}/actions-runner-linux-x64-${GITHUB_RUNNER_VERSION}.tar.gz | tar xz

registration_url="https://api.github.com/repos/${GITHUB_OWNER}/${GITHUB_REPOSITORY}/actions/runners/registration-token"
echo "Requesting registration URL at '${registration_url}'"

payload=$(curl -sX POST -H "Authorization: token ${GITHUB_PAT}" ${registration_url})
export RUNNER_TOKEN=$(echo $payload | jq .token --raw-output)

./config.sh \
    --name $(hostname) \
    --token ${RUNNER_TOKEN} \
    --url https://github.com/${GITHUB_OWNER}/${GITHUB_REPOSITORY} \
    --work ${RUNNER_WORKDIR} \
    --unattended \
    --replace

remove() {
    ./config.sh remove --unattended --token "${RUNNER_TOKEN}"
}

trap 'remove; exit 130' INT
trap 'remove; exit 143' TERM

./run.sh "$*" &

wait $!

However, because it appears the auto-updater wants to update to pre-release versions, I've modified it to not use the latest release, but the newest release:

GITHUB_RUNNER_VERSION=$(curl -s https://api.github.com/repos/actions/runner/releases| jq '.[0].tag_name' --raw-output | cut -c2-)

It would be quite nice to be able to disable the auto-updating. Will look into this eventually. Note that this hack is published as kubesail/github-actions:v19 if anyone else would like to make use of it :)

[myoung34]

Im unrelated to the project but I'm curious what makes 2.267.1 a pre-release. This was opened 10 days ago but 2.267.1 was released 19 days ago and doesn't show anything about being a pre-release afaict

I can try to work with you on this from my docker runner: https://github.com/myoung34/docker-github-actions-runner
by pinning to non-prerelease for specific tags

[npalm]

@myoung34 indeed the release was created 19 days ago, but it was marked as a pre-release till July the 15th or 16th. Unfortunately you cannot see anymore that the pre-released was promoted to a release.

[bryanmacfarlane]

The runner rolls out across rings. It's a service side config change and it's been tesedt, baked in all previous rings (including self host, github load, etc). So if it progressed beyond ring 0 into customer rings it's baked and not pre-release. My guess is someone didn't uncheck that box. @TingluoHuang

@erulabs as the related issues noted, run.sh should only be used for interactively running. In a container, you should call ./runsvc.sh. That gracefully handles updates.

[arash-bizcover]

@bryanmacfarlane How can I pass --once to the runsvc.sh ?

I'm using ./run.sh --once which help me have clean new containers after each build.

BTW on the latest release: v2.272.0 runsvc.sh is inside bin bin/runsvc.sh

[pietroalbini]

This happened again: 2.273.0 is being rolled out while being marked as a pre-release (we hit it in rust-lang-ci/rust).

[j3parker]

It would be nice to have some official guidance on this. E.g., "it is safe* to use the pre-release versions"

What we're looking to avoid is baking in an out-of-date version into our VMs. When we spin up to run a job the runner will sit idle with the out-of-date build until the first job comes in. It will accept that job, download the update, restart and then run the job. We are about to switch to disposable VMs which means that we will download the update for every job run.... we just want to avoid this latency.

Right now we poll the GitHub releases API hourly to get the download URL and pre-download the runner into our VM template (AWS AMI).

In a perfect world there would be some standard window of time we have to update. E.g. we can run the previous release or pre-release build for 24 hours/an hour/whatever before an auto-update (which would give us time to update our VMs)

[arash-bizcover]

Since Microsoft purchased Github ....