Quarkus 3.22: Test classes w/ superclasses fail with `IllegalAccessError`

[snazy]

Describe the bug

We're running into a class-loader issue in Quarkus 3.22.2 (worked fine w/ 3.19.3). Seems to be related to but doesn't look like the same as in #47692.

The issue happens for test classes that extend another class, something like this:

@QuarkusTest
@TestProfile(QuarkusTestProfilePersistInmemory.class)
class TestBasicOperations extends AbstractTestBasicOperations {

...

@ExtendWith(QuarkusNessieClientResolver.class)
abstract class AbstractTestBasicOperations {

TestBasicOperations + AbstractTestBasicOperations
Reproducible via ./gradlew :nessie-quarkus:test --tests TestBasicOperations --info against this branch

(leads to java.lang.IllegalAccessError: class org.projectnessie.server.TestBasicOperations cannot access its abstract superclass ...AbstractTestBasicOperations (...TestBasicOperations is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @9dbb1d9; org.projectnessie.server.AbstractTestBasicOperations is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @e4e1ef5)

If that test class doesn't have a superclass, it works fine. It happens for quite a bunch of tests.

It's also unrelated to our special ThreadPerTestClassExecutionExecutorService - same issue with the "vanilla JUnit engine/executor".

java.lang.ExceptionInInitializerError
java.lang.RuntimeException: java.lang.ExceptionInInitializerError
    at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:693)
    at io.quarkus.test.junit.QuarkusTestExtension.interceptTestClassConstructor(QuarkusTestExtension.java:791)
    at java.base/java.util.Optional.orElseGet(Optional.java:364)
    at org.projectnessie.junit.engine.ThreadPerTestClassExecutionExecutorService.executeTask(ThreadPerTestClassExecutionExecutorService.java:109)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
    at org.projectnessie.junit.engine.ThreadPerTestClassExecutionExecutorService.invokeAll(ThreadPerTestClassExecutionExecutorService.java:75)
    at org.projectnessie.junit.engine.ThreadPerTestClassExecutionExecutorService.lambda$executeTask$0(ThreadPerTestClassExecutionExecutorService.java:88)
    at java.base/java.lang.Thread.run(Thread.java:1575)
Caused by: java.lang.ExceptionInInitializerError
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:578)
    at java.base/java.lang.Class.forName(Class.java:557)
    at io.quarkus.runner.bootstrap.StartupActionImpl.run(StartupActionImpl.java:294)
    at io.quarkus.test.junit.QuarkusTestExtension.doJavaStart(QuarkusTestExtension.java:232)
    at io.quarkus.test.junit.QuarkusTestExtension.ensureStarted(QuarkusTestExtension.java:660)
    at io.quarkus.test.junit.QuarkusTestExtension.beforeAll(QuarkusTestExtension.java:712)
    ... 2 more
Caused by: java.lang.RuntimeException: Failed to start quarkus
    at io.quarkus.runner.ApplicationImpl.<clinit>(Unknown Source)
    ... 9 more
Caused by: java.lang.IllegalAccessError: class org.projectnessie.server.TestBasicOperations cannot access its abstract superclass org.projectnessie.server.AbstractTestBasicOperations (org.projectnessie.server.TestBasicOperations is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @9dbb1d9; org.projectnessie.server.AbstractTestBasicOperations is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @e4e1ef5)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1026)
    at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:565)
    at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:523)
    at org.projectnessie.server.TestBasicOperations_Bean.<init>(Unknown Source)
    at io.quarkus.arc.setup.Default_ComponentsProvider.addBeans9(Unknown Source)
    at io.quarkus.arc.setup.Default_ComponentsProvider.getComponents(Unknown Source)
    at io.quarkus.arc.impl.ArcContainerImpl.<init>(ArcContainerImpl.java:129)
    at io.quarkus.arc.Arc.initialize(Arc.java:38)
    at io.quarkus.arc.runtime.ArcRecorder.initContainer(ArcRecorder.java:52)
    at io.quarkus.runner.recorded.ArcProcessor$initializeContainer1770303700.deploy_0(Unknown Source)
    at io.quarkus.runner.recorded.ArcProcessor$initializeContainer1770303700.deploy(Unknown Source)
    ... 10 more

Also seeing another variant that's also related to class-loading for the Renovate-PR for Apache Polaris CI run

We suspect it's related to the test classloading rewrite.

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[snazy]

/cc @holly-cummins

[turing85]

FTR: https://quarkusio.zulipchat.com/#narrow/channel/187038-dev/topic/3.2E22.2E1.3A.20did.20something.20break.20with.20jacoco.3F/with/515542972

[gsmet]

@snazy could you identify the names of the two class loaders with a debugger?

[holly-cummins]

@snazy, I've just been looking at your reproducer, and I can reproduce with 3.22, but not with main. I'm a bit surprised by this since although there have been a lot of classloading fixes, none were specifically intended to address this problem. On the other hand, I haven't yet fully worked out the root cause, which makes it hard for me to say with authority what would or would not fix it.

I think 3.23.0.CR1 should be coming out today-ish, so maybe when it comes out you can try with it, and if you see it's working on your side too, we can declare victory. I'll also see if I can get a small enough reproducer to add as a unit test to prevent further regressions, although we have similar tests already that didn't catch this issue, so I'll need to figure out what's distinctive here.

[holly-cummins]

Ah, scratch that. After getting red builds with 3.22.2 and green builds with 999-SNAPSHOT, I then got green builds with 3.22.2 also. And then I went back to red builds. So either there's some intermittent behaviour or a cache is messing with me.

I'm also struggling to come up with a test case that reproduces without including all of Nessie; having a gradle project with @QuarkusTest with a profile, and an abstract superclass with @ExtendsWith a ParameterResolver isn't enough. But I don't know how much the non-reproducibility of my tests is messing with me.

[holly-cummins]

@snazy could you identify the names of the two class loaders with a debugger?

I've just brute-forced this, and for the record, the child is in the Runtime Classloader, and the parent is in the Base Runtime Classloader.

[holly-cummins]

Got it. This is specific to gradle, and only happens when the superclass is intestFixtures and is package-private. There doesn't need to be a profile or anything fancy on the superclass. The trivial workaround is to make the superclass public, but I'll work on a proper fix.

@ExtendWith(QuarkusNessieClientResolver.class)
public abstract class AbstractTestBasicOperations {

(Under the covers, because the parent class is in the test fixtures source, it can't get loaded by the runtime classloader, and so instead gets loaded by the base runtime classloader. The rule of package identity is the pair of package name + class loader identity, so the mismatched classloaders cause an access check failure.)