[4.0.x] Consolidate caches #11379
Merged
[4.0.x] Consolidate caches #11379
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: Our caches (all of them) are over 10GB allowed cache space. Due this, GH is dropping caches and our jobs end up exposed to HTTP 500 that since migration Central happens quite often. We used the plain `actions/cache` to store Mimir caches for each node (200-400 MB depending on which node we talk about), and we have 3 active branches (3.9, 4.0 and master), that simply totals out the 10 GB limit. This PR makes we have one "cache blob" per OS, so each OS has one cache blob (times three, for 3.9, 4.0 and master). Changes: * implement "always save" pattern (see cache doco) * we keep cache "per lane" (per OS) * 3 kind of builds (initial, full and integration-tests) all use same (per OS) cache at start and at end uploads cache as artifact (1 day retention) * at end there is a matrix job "consolidate caches" (runs on all 3 OSes) that downloads caches and consolidate them and save cache * hence, we will have 3 OS specific caches Backport of 304791e
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: Our caches (all of them) are over 10GB allowed cache space. Due this, GH is dropping caches and our jobs end up exposed to HTTP 500 that since migration Central happens quite often. We used the plain
actions/cacheto store Mimir caches for each node (200-400 MB depending on which node we talk about), and we have 3 active branches (3.9, 4.0 and master), that simply totals out the 10 GB limit.This PR makes we have one "cache blob" per OS, so each OS has one cache blob (times three, for 3.9, 4.0 and master).
Changes:
Backport of 304791e