Dev Services should be started during startup, not augmentation #45785
Description
One side effect of my "load test classes with the classloader used to run them" changes is that application augmentation and application start may now be separated in time. At the moment, we start dev services as part of augmentation. When augmentation and application start were butted up against each other, that seemed fine, but now they're separated, it doesn't feel quite right. We do all the discover up front, so we now do all the augmentation up front, which means we start dev services before they're needed.
This has several unfortunate consequences:
- If the tests in a test class are disabled or filtered out using tags, we still start the dev services. As discussed in QuarkusTest starts all dev services for all test profiles at startup, causing resource strain even when only a subset of tests are run #47771, this can cause excessive resource usage and means part of the benefit of disabling the test is lost.
- If two distinct @QuarkusTestResources share the same dev service, one of them will fail (bad!). We shut down the dev service when we stop the application, but we start the dev service during augmentation, not during startup so there's an asymmetry. The second test will think it can share a dev service with the first one, but it can't, because it will be shut down at the end of the first test. (Distinct resources sharing a dev service will cause test failures #45786)
- A similar issue, if one test in a suite uses a profile and the other one doesn't, and the test hardcodes a port for the dev service, the two services will try to start on the same port, and one will fail.
integration-tests/opentelemetry-redis-instrumentationuses this pattern. - If there is a catastrophic test failure, such as JUnit failing to discover tests (caused by an exception in the test discovery phase), containers will not be cleaned up
My first thought to fix this was to just make sure that the DockerStatusBuildItem wasn't produced until application start. That's pleasingly clean, but sadly doesn't work, because application start takes a BuildResult, so we can't be producing build items after that without totally changing the whole model.
Another option is to replicate the model. So as well as BuildItems, we could have StartupItems, which get chained in the same way BuildItems do, but which are for things that shouldn't happen during augmentation. A less radical option is to do some sort of partial-hacky-re-augmentation where something goes through BuildItems that should semantically be StartupItems and starts them. That's maybe not actually that different, in implementation, but it keeps the API the same so it's less invasive. So DockerStatusBuildItem could have an extra flag that says I actually am a runtime consideration and anything that consumes it then also gets deferred to runtime.
Lots of Quarkiverse extensions start dev services, so we can't just change stuff without doing a lot of checks.
@cescoffier points out that some of the config generated by the dev services may be build time properties. @radcortez says
I agree that DevServices should be runtime. We have a lot of issues, by trying to configure dev services with build time configuration, when what we need is the runtime configuration.
Build time properties can always be moved to build time runtime fixed. But not the other way around.
Another (maybe even bigger?) solution for this is to decouple discovery and test launch, which would mean writing a new JUnit engine. Discovery would happen in the normal way, and then we'd intercept before execution and create a new copy of the class. That avoids a whole bunch of eager augmentation problems, but isn't something that can be done with the current JUnit hooks.
--
On reflection, having experimented with a few options, I think the right tactical solution is just to not start dev services in augmentation, and start them during the startup phase. That sounds very simple, and the reality is a bit more complex, but I think it's both clean enough to be acceptable in the medium term, and quick enough to be a good tactical solution. The downside of this solution is that it needs to be applied to each dev service, so I'll make sub-issues to track the rollout.
We should also:
- Write a blog explaining what extension authors should do
- Update the writing dev services guide