Improve stability of e2e tests

[BacLuc]

Ways we could improve the stability of e2e tests

Improve the run environment

Thesis: The cypress github action is flaky, the docker images are better

Does not seem that way.

• run on upstream/devel with cypress github action: https://github.com/ecamp/ecamp3/actions/runs/9826683335
• run on BacLuc/e2e-use-docker-image: https://github.com/BacLuc/ecamp3/actions/runs/9826692413

Thesis: they are flaky because they don't have enough RAM

Does not seem that way

• run on upstream/devel with cypress github action: https://github.com/ecamp/ecamp3/actions/runs/9826683335
• run on BacLuc/e2e-restrict-ram: https://github.com/BacLuc/ecamp3/actions/runs/9826698872

Thesis: they are flaky because they don't have enough CPU

The feeling is that the tests would then just run longer and run into timeouts.
But this seems not the case.

Thesis: when we use the cypress retry mechanism, they will be more stable

It does not solve the problem, but renovate can merge and we can work.

• run on upstream/devel with cypress github action: https://github.com/ecamp/ecamp3/actions/runs/9826683335
• run on Bacluc/e2e-use-retry: https://github.com/BacLuc/ecamp3/actions/runs/9826705777

Thesis: the frankenphp container stops responding

(Which leads to the connection refused for the login request errors)
-> try enable strace: https://github.com/BacLuc/ecamp3/tree/debug-frankenphp-strace
add debug log to caddy: https://github.com/BacLuc/ecamp3/tree/enable-caddy-debug

-> this seems to be confirmed.
(See https://github.com/BacLuc/ecamp3/actions/runs/10013887138 in the failing runs in the step added by this PR: #5558)
I have 3 Solutions for this problem, they are listed in these 3 posts:

Vote with a 👍 or 👎

• Improve stability of e2e tests #5493 (comment)
• Improve stability of e2e tests #5493 (comment)
• Improve stability of e2e tests #5493 (comment)

Improve the single tests

The inspection of the test errors is in this Issue: #5322

Thesis: Not all tests are flaky

• Only clientPrint.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826741651
• Only httpCache.cy.js https://github.com/BacLuc/ecamp3/actions/runs/9826745947
• Only login.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826742342
• Only nuxtPrint.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826740641
• Only persistDashboardFilter.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826743440

[manuelmeister]

Thesis: they are flaky because browsers are flaky => use stable browser version
Thesis: they are flaky because Cypress is flaky => use Playwright?

[BacLuc]

Solution 1 for Thesis: "the frankenphp container stops responding"

Revert back to a classic php fpm setup

The thesis is, that the frankenphp is not yet production ready and crashes from time to time.
-> Then we should step back to the php fpm setup from before.

[BacLuc]

Solution 2 for Thesis: the frankenphp container stops responding

Use the prod frankenphp api image for the e2e tests

This seems to work (and hopefully so, we want to deploy this in prod).
Needs first #5277 (but is prepared)

Seems to work well: https://github.com/BacLuc/ecamp3/actions/runs/10014833334
But is a little complex:

• use api prod image for e2e tests #5560

still crashes, but less often: https://github.com/BacLuc/ecamp3/actions/runs/10014833334/job/27685355483

[BacLuc]

Solution 3 for Thesis: the frankenphp container stops responding

Add retries in cypress and restart the api container

This also seems to work. Firefox crashes a lot if you use a global beforeEach.
@BacLuc would like to remove firefox from the tests for now if we use this approach.

Seems to not work perfectly, but better as before (https://github.com/bacluc-test-org/ecamp3/actions/runs/9998897898):

• e2e: use retry for e2e tests #5561

We still did that, but with cypress retries in

• httpCache.cy.js: retry 'invalidates /camp/{campId}/categories for new category' in runMode #6260

[BacLuc]

Fix request and assertion order in httpCache.cy.js

• httpCache.cy.js: fix unstable 'invalidates /camp/{campId}/categories for new category' #6954
  or
• e2e: wait for the requests to either have cache 'HIT', 'MISS', 'PASS' #6953