Address flaky tests

[nathanjmcdougall]

I'm finding that tests\functional\test_error_handling.py::test_syntax_error_includes_module will fail when run on its own, but not when running the whole test suite.

On other projects I've had good success with detect-test-pollution which bisects the test suite to find leaky tests.

Basically you just run uv run detect-test-pollution --fuzz --tests ./tests and then follow the prompts.

Typical workflow looks like this:

Details

> uv run --with detect-test-pollution detect-test-pollution --fuzz --tests ./tests
discovering all tests...
-> discovered 852 tests!
run 1...
-> found failing test!
try `detect-test-pollution --failing-test tests/functional/test_build_and_use_graph.py::TestFindDownstreamModules::test_as_package_true --tests ./tests`!
>
> uv run --with detect-test-pollution detect-test-pollution --failing-test tests/functional/test_build_and_use_graph.py::TestFindDownstreamModules::test_as_package_true --tests ./tests
discovering all tests...
-> discovered 852 tests!
ensuring test passes by itself...
-> OK!
ensuring test fails with test group...
-> OK!
running step 1:
- 851 tests remaining (about 10 steps)
running step 2:
- 425 tests remaining (about 9 steps)
running step 3:
- 212 tests remaining (about 8 steps)
running step 4:
- 106 tests remaining (about 7 steps)
running step 5:
- 53 tests remaining (about 6 steps)
running step 6:
- 27 tests remaining (about 5 steps)
running step 7:
- 13 tests remaining (about 4 steps)
running step 8:
- 7 tests remaining (about 3 steps)
running step 9:
- 3 tests remaining (about 2 steps)
double checking we found it...
-> the polluting test is: tests/unit/adaptors/test_caching.py::TestCacheFileNamer::test_names_meta_cache_per_package

I plan to open a PR to:

• Add detect-test-pollution as a dev dependency
• Add just fuzz-test-pollution-python as a justfile entry
• Fix the flaky tests so that all subsets of the test suite pass

[nathanjmcdougall]

So, there was a bit of a misdiagnosis here:

• There are indeed polluting tests (as per the <details> section above).
• However, tests\functional\test_error_handling.py::test_syntax_error_includes_module isn't one of them. The reason it was failing is that it uses relative path access via .. and that can cause issues in some contexts (e.g. running via the VS Code GUI).
• The culprit for the test pollution seems to be the configure_unit_tests fixture. It isn't cleaning up after itself; it should be using the override_settings context manager