[GraalVM for JDK 24+] All native integration tests failing

[jerboaa]

Describe the bug

The last good revision is: ddf332b (which got built in our CI yesterday and was OK). Now - starting with revision ddd0ebc - all integration tests with Mandrel for JDK 24 (e.g. mandrel-java24-24.2.1.0-Final) fail at runtime with:

Error:  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.673 s <<< FAILURE! -- in io.quarkus.it.jpa.derby.JPAFunctionalityInGraalITCase
Error:  io.quarkus.it.jpa.derby.JPAFunctionalityInGraalITCase.testJPAFunctionalityFromServlet -- Time elapsed: 1.008 s <<< ERROR!
org.apache.http.NoHttpResponseException: localhost:8081 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:294)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:207)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:679)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:481)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at io.restassured.internal.RestAssuredHttpBuilder.doRequest(RestAssuredHttpBuilder.java:141)
	at io.restassured.internal.http.HTTPBuilder.doRequest(HTTPBuilder.java:496)
	at io.restassured.internal.http.HTTPBuilder.request(HTTPBuilder.java:453)
	at io.restassured.internal.http.HTTPBuilder$request$2.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:166)
	at io.restassured.internal.RequestSpecificationImpl.sendHttpRequest(RequestSpecificationImpl.groovy:1480)
	at io.restassured.internal.RequestSpecificationImpl.this$2$sendHttpRequest(RequestSpecificationImpl.groovy)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:790)
	at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:46)
	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:57)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at io.restassured.internal.RequestSpecificationImpl.sendRequest(RequestSpecificationImpl.groovy:1229)
	at io.restassured.internal.RequestSpecificationImpl.this$2$sendRequest(RequestSpecificationImpl.groovy)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:790)
	at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:46)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:166)
	at io.restassured.internal.filter.SendRequestFilter.filter(SendRequestFilter.groovy:30)
	at io.restassured.filter.Filter$filter.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at io.restassured.filter.Filter$filter$0.call(Unknown Source)
	at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
	at io.restassured.filter.time.TimingFilter.filter(TimingFilter.java:56)
	at io.restassured.filter.Filter$filter$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at io.restassured.filter.Filter$filter$0.call(Unknown Source)
	at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
	at io.restassured.filter.log.StatusCodeBasedLoggingFilter.filter(StatusCodeBasedLoggingFilter.java:112)
	at io.restassured.filter.log.ResponseLoggingFilter.filter(ResponseLoggingFilter.java:31)
	at io.restassured.filter.Filter$filter$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at io.restassured.filter.Filter$filter$0.call(Unknown Source)
	at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
	at io.restassured.filter.log.RequestLoggingFilter.filter(RequestLoggingFilter.java:148)
	at io.restassured.filter.Filter$filter$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at io.restassured.filter.Filter$filter.call(Unknown Source)
	at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
	at io.restassured.filter.FilterContext$next.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
	at io.restassured.internal.filter.CsrfFilter.filter(CsrfFilter.groovy:70)
	at io.restassured.filter.Filter$filter.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:157)
	at io.restassured.internal.filter.FilterContextImpl.next(FilterContextImpl.groovy:72)
	at io.restassured.filter.FilterContext$next.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
	at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1704)
	at io.restassured.internal.RequestSpecificationImpl.this$2$applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:790)
	at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:46)
	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:57)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:203)
	at io.restassured.internal.RequestSpecificationImpl.applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy:1710)
	at io.restassured.internal.RequestSpecificationImpl.this$2$applyPathParamsAndSendRequest(RequestSpecificationImpl.groovy)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:790)
	at io.restassured.internal.RequestSpecificationImpl.invokeMethod(RequestSpecificationImpl.groovy)
	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.call(PogoInterceptableSite.java:46)
	at org.codehaus.groovy.runtime.callsite.PogoInterceptableSite.callCurrent(PogoInterceptableSite.java:57)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:203)
	at io.restassured.internal.RequestSpecificationImpl.get(RequestSpecificationImpl.groovy:172)
	at io.restassured.internal.RequestSpecificationImpl.get(RequestSpecificationImpl.groovy)
	at io.quarkus.it.jpa.derby.JPAFunctionalityTest.testJPAFunctionalityFromServlet(JPAFunctionalityTest.java:18)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:836)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

To reproduce: Run a native integration test with Mandrel for JDK 24 (for example jpa).

See: https://github.com/graalvm/mandrel/actions/runs/15058004317/job/42328811735#step:12:496

All IT tests are red here (affects GraalVM CE build as well as Mandrel for JDK 24+). Mandrel for JDK 21 is OK.

[quarkus-bot]

/cc @Karm (native-image), @galderz (native-image), @zakkak (native-image)

[jerboaa]

Bisect tells me it's caused by #47618. FYI @cescoffier @Eng-Fouad

Edit: This time for real :)

[jponge]

Let's revert indeed

[jponge]

@jerboaa do you have any more precise log? Anything special to know about Graal on 24+? I'm asking because the native tests all passed on the regular Quarkus CI. Thanks!

[jerboaa]

@jerboaa do you have any more precise log? Anything special to know about Graal on 24+? I'm asking because the native tests all passed on the regular Quarkus CI. Thanks!

@jponge Nothing special. It doesn't show up with regular quarkus CI because that uses Mandrel for JDK 21 (this issue only shows up with JDK 24+ mandrel). The reproducers would be to just build native integration tests with a released Mandrel for JDK 24 (either a container image or a released tarball). For example use this one:
https://github.com/graalvm/mandrel/releases/tag/mandrel-24.2.1.0-Final

[jerboaa]

Reproducer could be this (my bisect script):

# Build quarkus
export JAVA_HOME="/disk/openjdk/builds/jdk-17.0.14+7/"
./mvnw clean install -Dquickly
EXIT_VAL=$?
if [ $EXIT_VAL -ne 0 ]; then
	exit 1
fi

# Run the integration test
export GRAALVM_HOME="/disk/graal/builds/mandrel-java24-24.2.1.0-Final"
export PATH="$PATH:$GRAALVM_HOME/bin"
podman system service --time=0 unix:/run/user/$(id -u)/podman.sock &
export DOCKER_HOST="unix:/run/user/$(id -u)/podman.sock"
SPECIFIC_TEST="jpa"
pushd integration-tests
../mvnw --fae -pl "$SPECIFIC_TEST" -Dnative -Dtest-containers -Dstart-containers -Dquarkus.native.enable-reports -Dnative.surefire.skip -Dformat.skip -Dno-descriptor-tests  -Dlog.level=ALL clean verify
EXIT_VAL=$?
popd
exit $EXIT_VAL

Exits with 1 on broken quarkus. Exits with 0 on good quarkus.

[jponge]

Sure, but if it worked with older GraalVM releases there must be something done differently that impacts the Netty substitution assumptions 😉

[jerboaa]

Sure, but if it worked with older GraalVM releases there must be something done differently that impacts the Netty substitution assumptions 😉

Yes. Apparently so.