Rewrite cassettes retry

[sckott]

fix #168

definitely squash and merge - lots of iterating to fix actions commits

Here's a run of this new workflow: https://github.com/FredHutch/wdl-unit-tests/actions/runs/14801625090/job/41732524382

Here's the generated PR from the above workflow run: #169

Changes here:

• now using pytest plugin pytest-json-report
• added fixture pytest_json_modifyreport that when the --json-report flag is used with pytest a report of test results is written to a json file. this fixture is not used when the --json-report flag is absent via this line @pytest.hookimpl(optionalhook=True)
• reformatted some make commands
• new make command test_api_rewrite_json_report to do cassette rewrites AND generate a json report
• modified the rewrite cassettes github action yaml to
  • generate the json report when testing
  • always succeed via exit 0 (otherwise any failures would not let us proceed)
  • open a PR with only the paths to cassettes and mock json files for tests that passed (failed test files are ignored)
  • then either Sean or I would merge the PR
  • the idea is for likely me to investigate any failures and fix those, then we can run rewrite cassettes again and merge that

¿Bueno?

[tefirman]

@sckott -- I like this addition, should make debugging much easier, but I'm a little confused where to look for the list of passed/failed test. I went to the GitHub Action linked above and the "List all cassettes passed" function just says "Array", same with failed cassettes. Am I looking in the wrong place?

[sckott]

@tefirman yeah I need to fix that. Its printing the class of the array of tests that failed or succeeded rather than the actual list

[sckott]

Waiting on fix to the test user from Dan

[sckott]

@tefirman okay, we're unblocked now by fix to test user, but the failed and succeeded tests are still not showing up correctly ...

[sckott]

Okay, @tefirman and @seankross can you take another look please - see last run https://github.com/FredHutch/wdl-unit-tests/actions/runs/14979902855

We're now doing the following approach:

• run test with make test_api_rewrite_json_report which runs tests and writes a report to a json file
• undo any changes to cassettes and mock json files associated with FAILED tests
• add only paths to the PR within the dirs: tests/cromwellapi/cassettes/* and tests/cromwellapi/mocked_submissions/*

With the current state of this action I haven't seen how it behaves when tests fail yet, but I'm pretty sure it will work well.

[seankross]

Thank you so much Scott, it looks clean but I can smell how this went through several painful iterations. I think this is right on the money though.

[seankross]

Love the elegance of make [command] || true

[sckott]

yeah, there's many ways to achieve this but this one seems good

[sckott]

thanks @seankross

[tefirman]

Looks awesome, really like the clear call-out of failing tests. Thanks for putting this together, @sckott !

[sckott]

thanks. @tefirman