Skip to content

feat(repository): use git bundles in runners + refactor credentials #605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 30 commits into
base: main
Choose a base branch
from

Conversation

corrieriluca
Copy link
Member

@corrieriluca corrieriluca commented May 23, 2025

This PR include changes from the following PRs:

And the following additional changes that merges the two:

  • Make repository controller use the GetChanges function from GitProvider to set LastRelevantCommit annotation on TerraformLayers
  • Test everything altogether
  • Write unit tests:
    • Webhook should annotate repositories on push events
    • RepositoryController should annotate layers when new revision is available
  • Enable by default the repository controller in Burrito's default config
  • Add documentation on migration to repository credentials
  • Add documentation on Git bundles and repository controller mechanics

List of tests implemented on the TerraformRepository controller

All tests pass and check if:

  • the TerraformRepository ends in the correct state
  • updates a git bundle to the datastore
  • annotates correctly the TerraformLayers it manages to trigger reconciliation on the TerraformLayer side (especially with the LastRelevantCommit annotation which is now required by the Layer controller to perform any action).

Nominal cases:

  • When a TerraformRepository without TerraformLayer is created
  • When a TerraformRepository has not TerraformLayers anymore
  • When a TerraformRepository with one TerraformLayer is created
  • When a TerraformRepository with multiple TerraformLayer is created
  • When a TerraformRepository has not been synced in the last 24h and changes are detected for some layers
  • When a new TerraformLayer is created for a existent TerraformRepository
  • When a TerraformRepository is already synced recently
  • When a TerraformRepository has not been synced in the last 24h but is already on last revision
  • When a TerraformRepository has a recent Sync Now annotation for a branch
  • When a TerraformRepository has an OLD Sync Now annotation for a branch

Error cases:

  • When the Git Providers fails for a newly created TerraformRepository
  • When the last sync of the TerraformRepository has already failed

@github-project-automation github-project-automation bot moved this to 📋 Backlog in burrito May 23, 2025
@codecov-commenter
Copy link

codecov-commenter commented May 23, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 28.01519% with 758 lines in your changes missing coverage. Please review.

Project coverage is 40.07%. Comparing base (ce980e1) to head (301ca65).

Files with missing lines Patch % Lines
internal/repository/providers/github/github.go 6.47% 129 Missing and 1 partial ⚠️
...ternal/repository/providers/standard/repository.go 0.00% 123 Missing ⚠️
internal/repository/providers/gitlab/gitlab.go 14.47% 64 Missing and 1 partial ⚠️
internal/repository/providers/mock/mock.go 0.00% 57 Missing ⚠️
internal/repository/providers/gitlab/api.go 0.00% 44 Missing ⚠️
internal/repository/credentials/credentials.go 66.40% 31 Missing and 11 partials ⚠️
internal/repository/providers/github/api.go 0.00% 41 Missing ⚠️
internal/datastore/client/client.go 0.00% 38 Missing ⚠️
internal/repository/repository.go 0.00% 37 Missing ⚠️
internal/datastore/client/mock.go 0.00% 35 Missing ⚠️
... and 15 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #605      +/-   ##
==========================================
+ Coverage   37.33%   40.07%   +2.74%     
==========================================
  Files          81       86       +5     
  Lines        5874     5771     -103     
==========================================
+ Hits         2193     2313     +120     
+ Misses       3522     3275     -247     
- Partials      159      183      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@corrieriluca corrieriluca force-pushed the feat/migrate-to-git-bundle branch 2 times, most recently from 0dafa71 to e5f6954 Compare May 30, 2025 12:07
@corrieriluca corrieriluca moved this from 📋 Backlog to 🏗 In progress in burrito May 30, 2025
@corrieriluca corrieriluca linked an issue Jun 1, 2025 that may be closed by this pull request
@corrieriluca corrieriluca force-pushed the feat/migrate-to-git-bundle branch from 9978e89 to 12390b1 Compare June 13, 2025 12:26
@Laudenlaruto Laudenlaruto removed this from burrito Jun 20, 2025
@corrieriluca corrieriluca marked this pull request as ready for review June 20, 2025 15:57
@corrieriluca corrieriluca force-pushed the feat/migrate-to-git-bundle branch from ad1952a to 27befce Compare June 27, 2025 14:41
@corrieriluca
Copy link
Member Author

Documentation is now up-to-date with changes in the PR

corrieriluca and others added 16 commits July 5, 2025 17:07
* fix(repo-controller): fix typos in some logs

* feat(repo): add annotation on layers for new revisions

* feat(run): wip: launch on specific bundle revision

* fix(datastore): handle not found error for S3 HeadObject

* feat(run): clone on bundles

* fix(repo-controller): add errors in warning events

* feat(repo-controller): trigger sync on new layer creation

* test(layer): fix tests with last branch commit annotation

* test(run): fix tests with the revision bundles

* test(runner): fix tests with usage of git bundles
…epository provider (#528)

* refactor(provider): create dedicated credentials package and rework repository provider

* test(credentials): cover most non transient paths

* fix(creds): ttl condition on updating credentials

* fix(creds): add some comments in credential store

* fix(credentials): make mock provider work

* feat(credentials): controllers use credentials store

* feat(webhook): webhook server use credentials store

* feat(manifests): remove secretName from tfrepo

* refactor(config): remove deprecated structs for credentials

* fix(lint): check err values

* fix(test): rework test NormalizeURL

* fix(lint): check err values

* feat(config): add config key for credentials TTL

* feat(repo-controller): update bundle with latest changes

* fix(credentials): add secret indexer on field type

* fix(providers): use the correct repo URL

---------

Co-authored-by: Alan <[email protected]>
Co-authored-by: Luca Corrieri <[email protected]>
Co-authored-by: Lucas Marques <[email protected]>
@corrieriluca corrieriluca force-pushed the feat/migrate-to-git-bundle branch from 27befce to 301ca65 Compare July 5, 2025 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat(repository): implement repository caching
3 participants