Test rewrites fixes

[sckott]

The scheduled rewrites are inconsistent - fixme

[sckott]

I've tried (commit above 606316e) retrying the entire test suite if ANY errors with https://github.com/nick-fields/retry

This hasn't worked so far. I've tried running the action maybe 4 times and each run there's 3 attempts (we could do more, but 3 seems reasonable) and different errors on each attempt.

The issue here comes back to I think just the variation in what happens on rhino/gizmo

Upon failures in the checks above, I can never recreate them locally - suggesting that they are intermittent - so there could be fixes to the tests but it's been difficult to reproduce errors

[sckott]

I tried checking for test idempotency with https://github.com/tyleryep/pytest-idempotent but it didn't work / couldn't get it to work

[sckott]

Next I'm going to try letting the rewrite action proceed with any failures and simply ignore cassette changes for tests that failed, then open the PR and we can merge.

This way we have updated cassettes for likely most tests.

Then we can look at any failed tests and update those cassettes manually and/or fix those tests if they need fixing.

[sckott]

The above is kinda complicated, so I'm thinking next to try recording probably in a json file the tests that succeed and those that fail and then use those paths to pass to git commit ...

There's a few options:

• using pytest-json-report package with command line flags like --json-report and --json-report-file=results.json