TST: `yt/visualization/volume_rendering/tests/test_scene.py::test_annotations` is failing in CI

[neutrinoceros]

Bug report

The following test is currently failing in CI, however I couldn't reproduce the error locally yt/visualization/volume_rendering/tests/test_scene.py::test_annotations

see https://github.com/yt-project/yt/actions/runs/7626456177/job/20772882827

[neutrinoceros]

Notes:

• the test still passes in minimal env tests
• the fact that we haven't been able to reproduce it locally (by running just this test) may be indicative of test pollution, which could be checked by running only this test in CI too.

[chrishavlin]

the fact that we haven't been able to reproduce it locally (by running just this test) may be indicative of test pollution, which could be checked by running only this test in CI too.

I just confirmed this over in #4786:

• running just pytest yt/visualization/volume_rendering/tests/test_scene.py::test_annotations passed in CI
• expanding to run all the visualization tests, pytest yt/visualization/, passed in CI

[chrishavlin]

was trying to re-run the failed test with pytest --lf as well to see if it'd fail on the second go, but i couldn't get it working in the github action...

might also be worth doing a run that randomizes test order?

[neutrinoceros]

might also be worth doing a run that randomizes test order?

Ideally yes, though I suspect that would be very unstable if we did it now because pollution in our test is possibly high, and it's likely that some of them are only passing because of other tests' side effects (I've put a lid on this for a while ...).

[chrishavlin]

ok, well i've narrowed down the subset of tests that you need to run to get the failure: just running the tests in yt/data_objects/tests and the failing test:

pytest --color=yes yt/data_objects/tests yt/visualization/volume_rendering/tests/test_scene.py::test_annotations

no indication of why yet. Also, when re-running the failed test, it does pass on second run.

[chrishavlin]

very close to an answer! I used the https://github.com/asottile/detect-test-pollution pytest plugin to find that yt/data_objects/tests/test_firefly.py::test_firefly_JSON_string is the source of the issue. Running just these two tests you can reproduce the failure:

        pytest --color=yes  \
        yt/data_objects/tests/test_firefly.py::test_firefly_JSON_string \
        yt/visualization/volume_rendering/tests/test_scene.py

when reversing the order, tests pass no problem. Looking at test_firefly_JSON_string, there's nothing obvious... but those firefly reader objects seem to do a lot with writing to disk and symlinks and default data directories so maybe there's something there?

[neutrinoceros]

Great job ! I may be able to give a second read to those tests as early as tomorrow. Meanwhile: friendly ping @agurvich

[chrishavlin]

OK. I think i'm done for the day, but one extra note: I still get the failure when I change test_firefly_JSON_string to not actually call writeToDisk, so it's something in the initial creation of the firefly object in ad.create_firefly_object (could be a yt thing, a firefly thing or a shared library between the two or something else very strange? really still no good idea...)

[agurvich]

but those firefly reader objects seem to do a lot with writing to disk and symlinks and default data directories so maybe there's something there?

don't talk about my trash like that! hahaha. I don't have a working yt development environment set-up, but it sounds like I might need to sniff something out?

The situation with the symlinks is that the firefly data needs to be written out to disk in order to be read in by the browser (unless it's passed through a websocket using flask). However, because of browser security features, websites (or webapps) can't read files just anywhere on disk, they have to be subdirectories of the app itself. So when you're hosting this app locally on your computer, the data has to be visible from a path that's a subdirectory of the app files (which int he case of being installed via pip, is hidden in your site-packages directory). My solution at the time was to let users specify where they wanted to write their data to and then symlink from there to Firefly's data directory. So what yt should be doing here is creating a Firefly object, passing some array data to it, using the Firefly object's built-in writeToDisk to write the data to a named temporary file, and then make a symlink from there to Firefly's data directory. I don't totally remember how I set up the unit test and I'm not sure what is actually failing here (is there a failing test or is there an incomprehensible error?). Where can I see the CI output (without building and running locally myself)?

Or will I have to build yt?

[chrishavlin]

don't talk about my trash like that! hahaha.

haha, many apologies :D I don't actually think that the symlinks are the culprit, I was just grasping at straws cause I'm rather baffled by this failure.

Here's a quick summary of the issue:

• a test unrelated to firefly started failing recently (yt/visualization/volume_rendering/tests/test_scene.py::test_annotations). neither myself or @neutrinoceros have been able to reproduce this issue locally, it only fails in our CI pipeline on github with python 3.12.
• we figured out it a test pollution issue: re-running the failed test by itself passes just fine
• I bisected down to the point where I can re-create the failure with just two tests: a yt test that creates a firefly object (yt/data_objects/tests/test_firefly.py::test_firefly_JSON_string) and our test_annotations test. It only fails when the firefly related test is run before the test_annotations test, reversing the order passes. So some global state somewhere is somehow being changed? maybe??

here's an example of the failing test that reads back in an image that was written to disk.