Tests on CI run slow, too often and flaky

[usu]

This is a collection of issues I see with our current CI setup that make development slow and makes the life of renovate hard.

Problem description

• Generally, our tests take a rather long time (~15min in average)
• The tests run often, sometimes testing the same commit that was already tested before
• Our e2e tests are still flaky, leading to necessary repetition of tests
  • Higher utilization of workflow runners
  • PRs getting kicked out of merge queue (=queue has to restart other jobs)
  • Renovate not being able auto-merge => high clutter in PR backlog
• Fixing a bug and deploying it to prod takes a huge time (as we saw during the last hackathon). CI needs to run min. 6 times successfully:
  • First, CI needs to run on PR (twice in parallel)
  • Then CI needs to run on merge queue
  • After merging a PR from merge queue, dev again runs CI (although the same commit was already tested in merge queue)
  • When fast-forwarding to stage & prod, the CI runs again (although the same commit was already tested before)
• Renovate still seems to have a problem with our setup: Renovate mixing different upgrades in same PR and even downgrades packages #4309
  This might be unrelated to above. I couldn't figure out anything on renovatebot issues list. I wonder though if this is somehow connected to merge queue and the config rebaseWhen:conflicted.

Potential solutions (to be discussed)

• Challenge benefit of merge queue (does it bring the benefits we hoped?)
• Making e2e tests non-mandatory tests? Or at least for renovate? Yes, this means we could potentially break dev. I don't think though that I remember a dependency upgrade only breaking e2e and no other tests.
• Somehow making API tests faster
• When pushing to a pull request, the CI runs twice (1x for push, 1x for pull request). Do we really need both?
• Find a way to avoid running the same CI for a commit hash where CI has already run successfully before
• other ideas?

[manuelmeister]

Challenge benefit of merge queue (does it bring the benefits we hoped?)

I quite like it. Especially when many people are actively pushing/merging

Making e2e tests non-mandatory tests? Or at least for renovate? Yes, this means we could potentially break dev. I don't think though that I remember a dependency upgrade only breaking e2e and no other tests.

I think we should try to make the e2e test less flakey. I don't like not testing them, as before we upgrade to vue3 I'd like add more tests. We could try to use https://docs.cypress.io/guides/continuous-integration/github-actions#Testing-with-Cypress-Docker-Images instead of pulling the latest one.

Somehow making API tests faster

The question is, are we testing the right way?

When pushing to a pull request, the CI runs twice (1x for push, 1x for pull request). Do we really need both?

That was once the plan.

Find a way to avoid running the same CI for a commit hash where CI has already run successfully before

We could try to run e.g. the api test on PR push only if certain conditions match?

[manuelmeister]

Core Meeting Decision

Things we want to tackle:

• Try to use cypress image CI: use image in docker compose again to run e2e tests #5414
• Improve Stability of cypress tasks
• Improve Stability of api tests (fix api performance_test #5292)
• Reduce duration of api tests (maybe parallelism)
  • Happened without our doing: Destroy TestCase object after its test was run sebastianbergmann/phpunit#5861
  • Further reduction of CI time: Use paratest for parallel test execution #5372
• Increase the duration of the performance check. We would like to have a benchmark, because the CI is not guaranteed to run stable or performant
  • Create reporting of certain dev endpoints under repeatable circumstances Currently we are more or less good with monitoring the performance in prod, testing against prod data locally and checking the performance on staging.
• Local E2E use separate DB
  • -> Reduce errors by running the e2e tests against different database #7909

Maybe/explore:

• Only run stable e2e tests
• skip duplicate actions

[BacLuc]

Might be closed except: "Local E2E use separate DB"

[BacLuc]

Closed with follow up Issue.