Native compilation fails on unreachable symbols

[mocenas]

Describe the bug

Compiling quarkus app into native image occasionally fails on Error: The heap snapshot verifier discovered a type not marked as reachable. see example
This happens in testSuite and it happens randomly - on different application sometimes on amd64 architecture sometimes on aarch64.
I cannot provide any specific way how to reproduce it, as this seems to happen randomly. But we're experiencing this in our daily tests every day since March 14 2025.

Examples of failures:

04:19:32,806 INFO  mvn: Error: The heap snapshot verifier discovered a type not marked as reachable: AnalysisType<HotSpotDiagnosticMXBean$ThreadDumpFormat[] -> HotSpotType<[Lcom/sun/management/HotSpotDiagnosticMXBean$ThreadDumpFormat;, resolved>, allocated: false, inHeap: false, reachable: false>
04:19:32,806 INFO  mvn: com.oracle.graal.pointsto.util.AnalysisError: The heap snapshot verifier discovered a type not marked as reachable: AnalysisType<HotSpotDiagnosticMXBean$ThreadDumpFormat[] -> HotSpotType<[Lcom/sun/management/HotSpotDiagnosticMXBean$ThreadDumpFormat;, resolved>, allocated: false, inHeap: false, reachable: false>
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.guarantee(AnalysisError.java:184)
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.heap.HeapSnapshotVerifier$ScanningObserver.ensureTypeScanned(HeapSnapshotVerifier.java:332)
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.heap.HeapSnapshotVerifier$ScanningObserver.forScannedConstant(HeapSnapshotVerifier.java:318)
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.scanConstant(ObjectScanner.java:252)
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.scanField(ObjectScanner.java:181)
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.doScan(ObjectScanner.java:386)
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.lambda$scanConstant$6(ObjectScanner.java:257)
04:19:32,806 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
04:19:32,807 INFO  mvn: 	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
04:19:32,807 INFO  mvn: 	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
04:19:32,807 INFO  mvn: 	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
04:19:32,807 INFO  mvn: 	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
04:19:32,807 INFO  mvn: 	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
04:19:32,807 INFO  mvn: 	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
04:19:32,808 INFO  mvn: 	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

mvn: com.oracle.graal.pointsto.util.AnalysisError: The heap snapshot verifier discovered a type not marked as reachable: AnalysisType<MemoryType[] -> HotSpotType<[Ljava/lang/management/MemoryType;, resolved>, allocated: false, inHeap: false, reachable: false>

mvn: com.oracle.graal.pointsto.util.AnalysisError: The heap snapshot verifier discovered a type not marked as reachable: AnalysisType<HotSpotDiagnosticMXBean$ThreadDumpFormat[] -> HotSpotType<[Lcom/sun/management/HotSpotDiagnosticMXBean$ThreadDumpFormat;, resolved>, allocated: false, inHeap: false, reachable: false>

mvn: com.oracle.graal.pointsto.util.AnalysisError: The heap snapshot verifier discovered a type not marked as reachable: AnalysisType<MemoryType[] -> HotSpotType<[Ljava/lang/management/MemoryType;, resolved>, allocated: false, inHeap: false, reachable: false>

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

Mandrel or GraalVM version (if different from Java)

No response

Quarkus version or git rev

999-SNAPSHOT

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @Karm (native-image), @galderz (native-image), @zakkak (native-image)

[galderz]

@mocenas This might be a bit tricky to pin down. Do you have any clues on what might have changed on March 14 2025 (or slightly before) to start causing this? A Quarkus update? A Mandrel update? That might give us some clues on what has caused it.

[mocenas]

@mocenas This might be a bit tricky to pin down. Do you have any clues on what might have changed on March 14 2025 (or slightly before) to start causing this? A Quarkus update? A Mandrel update? That might give us some clues on what has caused it.

These test use quarkus daily snapshot (if you count that as quarkus update). Otherwise I found nothing significant that changed on our side. We've bumped quarkus-ide-config version, but doesn't seem to me, that it should affect this.

I found this issue happens on build images using both mandrel and graalvmce. We use images tagged ":jdk-21". So it is possible that new image was made, but can't really say that.

I tried to look into some quarkus PRs that might have caused this, but nothing really appears to me. I found some that might have some connection to this, but it is really just wild guess:

• Update Maven Compiler Plugin to 3.14.0 #46757
• UBI9 microdnf requires -y #46763

[galderz]

@mocenas Any changes in https://github.com/quarkus-qe/quarkus-test-suite around the 14th of March that could have caused this?

These test use quarkus daily snapshot (if you count that as quarkus update)

Daily snapshot of which quarkus branch? Main?

@Karm @zakkak @jerboaa could an image update around the 14th of March could have started to cause this? I'm not aware of any functionality change around that area. The last CPU was done in January.

[mocenas]

Daily snapshot of which quarkus branch? Main?

Yes daily snapshots of quarkus main - version "999-SNAPSHOT"

[gsmet]

I would try to copy this test and try to reduce the number of extensions (for instance dropping the DB2 driver or the Oracle driver) and see how it goes.

That might help pinpointing the culprit code.

I had a look at what was merged to Quarkus main on March 14th and I don't see anything that could be problematic (but then these issues are odd so you never know).

[jerboaa]

@mocenas I've noticed that this is using a GraalVM CE builder image (JDK is 21.0.2 - which is very old). Is this reproducible with the latest Mandrel 23.1 image as well?

[mocenas]

@mocenas I've noticed that this is using a GraalVM CE builder image (JDK is 21.0.2 - which is very old). Is this reproducible with the latest Mandrel 23.1 image as well?

This problem is not really reliably reproducible. I've already encountered this problem with both ubi9-quarkus-graalvmce-builder-image:jdk-21 and ubi9-quarkus-mandrel-builder-image:jdk-21. I will try it it with mandrel ubi9-quarkus-mandrel-builder-image:23.1-jdk-21 image and will see.

[mocenas]

Switching to mandrel 23.1-jdk-21. Seems to help, cannot reproduce the issue with it. But mandrel 21 is AFAIK still supported for quarkus, so we cannot simply say "use newer image". I'm I right?

This issue was already discussed on graal oracle/graal#10081 and also on zulip https://quarkusio.zulipchat.com/#narrow/channel/187030-users/topic/quarkus.203.2E19.2E1.20.3E.20native.20build.20fails

[jerboaa]

Switching to mandrel 23.1-jdk-21. Seems to help, cannot reproduce the issue with it.

Great. That's good to know.

But mandrel 21 is AFAIK still supported for quarkus, so we cannot simply say "use newer image". I'm I right?

What do you mean by Mandrel 21 (some old JDK 11/17 based image)? If so, that's not true. It's no longer supported. The only Mandrel image currently supported for quarkus is "Mandrel for JDK 21" and newer (that is, Mandrel for JDK 24 currently only - to be released in the following days).

This issue was already discussed on graal oracle/graal#10081 and also on zulip https://quarkusio.zulipchat.com/#narrow/channel/187030-users/topic/quarkus.203.2E19.2E1.20.3E.20native.20build.20fails

So if this is an issue with an old, no longer supported version, we should close this. To sanity check you can try with a GraalVM for JDK 24 build here.

[mocenas]

Sorry for confusion about version numbers here. I meant mandrel for java 21 not mandrel 21.
The ubi9-quarkus-mandrel-builder-image:23.1-jdk-21 image is actually the same image as ubi9-quarkus-mandrel-builder-image:jdk-21 Both of them are mandrel 23.1 (verified in logs, they use same mandrel version and both of those images have same checksum, so they are really same image).

To recap, this issue happens on mandrel 23.1 image with java 21. Switching to image 23.1-jdk-21 does not actually help (since it uses same image as before), was probably just luck, that I did not experience it with.

So issue still stands.

[jerboaa]

Does it happen with ubi9-quarkus-mandrel-builder-image:jdk-23 or ubi9-quarkus-graalvmce-builder-image:jdk-23? If we know newer images are OK, we "just" need to find the right fix for a backport.

[mocenas]

Does it happen with ubi9-quarkus-mandrel-builder-image:jdk-23 or ubi9-quarkus-graalvmce-builder-image:jdk-23? If we know newer images are OK, we "just" need to find the right fix for a backport.

I will check and let you know.