Skip to content

Conversation

@jamescrosswell
Copy link
Collaborator

@jamescrosswell jamescrosswell commented Nov 17, 2025

Resolves #4788

The containers are pretty lean and I didn't see any opportunity to trim these down further. This PR cleans up a few temporary files before the tests are run (as that's when the jobs were failing). It doesn't work miracles but probably saves enough space to keep everything running a little longer:

Run echo "Disk usage before cleanup:"
Disk usage before cleanup:
Filesystem                Size      Used Available Use% Mounted on
overlay                  71.6G     70.0G      1.6G  98% /
...
Clearing NuGet HTTP cache: /github/home/.local/share/NuGet/http-cache
Local resources cleared.
Disk usage after cleanup:
Filesystem                Size      Used Available Use% Mounted on
overlay                  71.6G     69.1G      2.5G  96% /
...

See also

#skip-changelog

@jamescrosswell jamescrosswell changed the title Try reintroducing free disk space task for containers fix: System.IO.IOException: No space left on device on linux-musl x64 Nov 17, 2025
@codecov
Copy link

codecov bot commented Nov 17, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.91%. Comparing base (64b0df9) to head (7970193).
⚠️ Report is 45 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4743      +/-   ##
==========================================
+ Coverage   73.66%   73.91%   +0.25%     
==========================================
  Files         476      485       +9     
  Lines       17442    17682     +240     
  Branches     3453     3494      +41     
==========================================
+ Hits        12848    13070     +222     
- Misses       3746     3755       +9     
- Partials      848      857       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jamescrosswell
Copy link
Collaborator Author

jamescrosswell commented Nov 17, 2025

The Alpine container that we were using was the nightly build. Up until recently, that had .net 10 rc2 on it. Recently this must have been updated to include the GA release instead.

Our workflows install the version of .net that we have pinned and up until recently that was rc2... so when the Alpine container bumped to GA, we ended up with two versions of .net on there: rc2 and ga. After merging #4723, the problem seems to have been resolved.

edit: spoke too soon... this has started breaking again.

@Flash0ver Flash0ver deleted the disk-space-musl branch November 17, 2025 10:51
@jamescrosswell jamescrosswell restored the disk-space-musl branch December 8, 2025 23:40
@jamescrosswell jamescrosswell reopened this Dec 8, 2025
@jamescrosswell jamescrosswell marked this pull request as ready for review December 9, 2025 20:39
build.binlog
if-no-files-found: ignore

- name: Free disk space before tests
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Cleanup action runs inside container, not on runner

The condition if: ${{ !matrix.container }} was removed from the "Remove unused applications" step. This causes the freediskspace action to run inside Alpine containers rather than being skipped. Since the PR aims to free space on the runner (not the container), running cleanup inside the container won't achieve the intended goal. Additionally, the freediskspace action uses Ubuntu-specific commands (apt-get) that don't exist in Alpine Linux (which uses apk). The original condition correctly prevented this action from running when a container is specified.

Fix in Cursor Fix in Web

Comment on lines +50 to +56
# Clean up apt packages
sudo apt-get remove -y '^mysql-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^mysql-.*' --fix-missing] failed to complete successfully. Proceeding..."
sudo apt-get remove -y '^postgresql-.*' --fix-missing || echo "::warning::PostgreSQL removal failed"
sudo apt-get remove -y '^mongodb-.*' --fix-missing || echo "::warning::MongoDB removal failed"
sudo apt-get autoremove -y || echo "::warning::The command [sudo apt-get autoremove -y] failed to complete successfully. Proceeding..."
sudo apt-get clean || echo "::warning::The command [sudo apt-get clean] failed to complete successfully. Proceeding..."
df -h /dev/root
Copy link
Member

@Flash0ver Flash0ver Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: We get these warnings during the build:

# .NET (linux-musl-x64)
The command [sudo apt-get clean] failed to complete successfully. Proceeding...
# .NET (linux-musl-x64)
The command [sudo apt-get autoremove -y] failed to complete successfully. Proceeding...
# .NET (linux-musl-x64)
MongoDB removal failed
# .NET (linux-musl-x64)
PostgreSQL removal failed
# .NET (linux-musl-x64)
The command [sudo apt-get remove -y '^mysql-.*' --fix-missing] failed to complete successfully. Proceeding...

# .NET (linux-musl-arm64)
The command [sudo apt-get clean] failed to complete successfully. Proceeding...
# .NET (linux-musl-arm64)
The command [sudo apt-get autoremove -y] failed to complete successfully. Proceeding...
# .NET (linux-musl-arm64)
MongoDB removal failed
# .NET (linux-musl-arm64)
PostgreSQL removal failed
# .NET (linux-musl-arm64)
The command [sudo apt-get remove -y '^mysql-.*' --fix-missing] failed to complete successfully. Proceeding...

Can we if-Condition to only run on supported platforms / not run on non-supported platforms (linux-musl-x64 and linux-musl-arm64)?
Maybe by splitting it into a step that does sudo rm, and another step that does the sudo apt-get (but not for Alpine Linux)?

Maybe something like:

- name: Remove unused packages
  if: ${{ !contains(matrix.container.image, 'alpine') }}
  run: |
    # Clean up apt packages
    sudo apt-get ...
    ...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's pretty much what we had before:
image

I'll put that bit back since the freediskspace action is ineffective on the containers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I meant to just execute thesudo apt-get conditionally, since they produce the ::warning:: messages.
But keep the sudo rm on all Linux runners, since they seem to work just fine.

Copy link
Collaborator Author

@jamescrosswell jamescrosswell Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah OK, we can tidy up the non-container runners in another PR if need be 👍🏻

@jamescrosswell jamescrosswell merged commit b186caa into main Dec 10, 2025
17 checks passed
@jamescrosswell jamescrosswell deleted the disk-space-musl branch December 10, 2025 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integration tests are flaky on linux-musl-x64 runners

3 participants