Rest client HTTP2 issue in Quarkus 3.20.2.2 and later

[shuklarakeshk]

Describe the bug

When utilizing Quarkus 3.20.2, the REST client successfully establishes an HTTP/2 connection to the OpenShift REST endpoint, with HTTP/2 functionality enabled through the configuration parameter quarkus.rest-client.http2=true. However, upon upgrading to Quarkus version 3.20.2.2 or subsequent releases, the identical implementation fails and gives 500 error

Expected behavior

The REST client should successfully establish a connection and retrieve data from the OpenShift API REST endpoint, maintaining the same connectivity behavior as observed in the Quarkus 3.20.2 platform version.

Actual behavior

When using Quarkus Platform 3.20.2.2 and later it is failing with below exception

Error id 7036361c-6487-471e-b74f-2faba199e1b4-2, java.io.IOException: Stream was closed (GOAWAY error code = 0)
Stack:
	java.io.IOException: Stream was closed (GOAWAY error code = 0)
	at org.jboss.resteasy.reactive.client.handlers.ClientSendRequestHandler$4.handle(ClientSendRequestHandler.java:402)
	at org.jboss.resteasy.reactive.client.handlers.ClientSendRequestHandler$4.handle(ClientSendRequestHandler.java:393)
	at io.vertx.core.impl.future.FutureImpl$2.onFailure(FutureImpl.java:117)
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onFailure(FutureImpl.java:316)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278)
	at io.vertx.core.http.impl.HttpClientRequestBase.fail(HttpClientRequestBase.java:186)
	at io.vertx.core.http.impl.HttpClientRequestBase.handleException(HttpClientRequestBase.java:181)
	at io.vertx.core.http.impl.HttpClientRequestImpl.handleException(HttpClientRequestImpl.java:90)
	at io.vertx.core.http.impl.Http2ClientConnection$StreamImpl.handleException(Http2ClientConnection.java:557)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:342)
	at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:163)
	at io.vertx.core.http.impl.VertxHttp2Stream.onException(VertxHttp2Stream.java:97)
	at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:153)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler$1.onStreamClosed(VertxHttp2ConnectionHandler.java:93)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:355)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1043)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:999)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:521)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.doCloseStream(Http2ConnectionHandler.java:919)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:627)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.closeStreamRemote(Http2ConnectionHandler.java:619)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:323)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:409)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:244)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:164)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186)
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61)
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelRead(VertxHttp2ConnectionHandler.java:405)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)
* Connection #0 to host 127.0.0.1 left intact

How to Reproduce?

Reproducer (zip file attached)
Zip file contains simple Quarkus based application with one exposed REST endpoint and one REST client

reproducer.zip

Environment:
Tested with CRC (OpenShift local). The same behavior occurs on OpenShift 4.16.

How to run the reproducer:

Start OpenShift:
    crc: `crc start`  Or use any OpenShift 4.16 API server URL.
Wait for the cluster to be ready. If using an existing OpenShift cluster, proceed to the next step.
Start the application: `./gradlew quarkusDev`
Exercise the endpoint: `curl -v http://127.0.0.1:8080/test-openshift`

Expected (failing) result:
The request should return HTTP 500 and produce the exception trace.

How to verify with a working Quarkus version:

In gradle.properties, uncomment the lines that set the Quarkus platform version to 3.20.2.
If necessary, switch the Gradle wrapper to Gradle 8.x: in gradle-wrapper.properties, uncomment the Gradle 8.x distribution line and comment the Gradle 9.x line.
Re-run: ./gradlew quarkusDev
Call the endpoint again: curl -v http://127.0.0.1:8080/test-openshift

Expected (working) result:
The endpoint should return a JSON response.

Output of uname -a or ver

Linux sonata 6.16.8-200.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 19 17:47:18 UTC 2025 x86_64 GNU/Linux

Output of java -version

java 21 2023-09-19 LTS Java(TM) SE Runtime Environment (build 21+35-LTS-2513) Java HotSpot(TM) 64-Bit Server VM (build 21+35-LTS-2513, mixed mode, sharing) sonuka@sonata:~/tools/crc$

Quarkus version or git rev

3.28.1

Build tool (ie. output of mvnw --version or gradlew --version)

9.1.0

Additional information

As the application works with Quarkus 3.20.2, it fails with Quarkus 3.20.2.2 and later. I discovered that the Vert.x Core JAR is causing this issue. When upgrading Quarkus to 3.20.2.2, for example, forcing the use of io.vertx:vertx-core:4.5.17 resolves the problem.

vertx-core diff

[quarkus-bot]

/cc @cescoffier (rest-client), @geoand (openshift,rest-client), @iocanel (openshift)

[geoand]

cc @vietj

[cescoffier]

It most probably because of the made you reset cve fix.

[cescoffier]

Note that I'm unable to reproduce it with:

Server Version: 4.19.7
Kubernetes Version: v1.32.6

I can't install CRC on my machine (wrong architecture)

[geoand]

I assume the same problem happens with the latest versions of Quarkus, right?

[shuklarakeshk]

Note that I'm unable to reproduce it with:

Server Version: 4.19.7
Kubernetes Version: v1.32.6

I can't install CRC on my machine (wrong architecture)

ok, I tested it with Openshift which has K8S version 1.29.11+x , ironically curl is able to connect with crc and openshift server when running command

curl -v --http2 https://api.crc.testing:6443/.well-known/oauth-authorization-server

or

curl -v --http2 --http2-prior-knowledge https://api.crc.testing:6443/.well-known/oauth-authorization-server

[shuklarakeshk]

I assume the same problem happens with the latest versions of Quarkus, right?

yes , but I have tested it against Openshift with K8S version 1.29.11+x

[cescoffier]

I tried with another OpenShift cluster (@rsvoboda can give the details) - no problem. The connection worked flawlessly.

[shuklarakeshk]

I tried with another OpenShift cluster (@rsvoboda can give the details) - no problem. The connection worked flawlessly.

Now, I am uncertain whether I am encountering an issue due to my own setup or a systemic problem. While analyzing tcpdump, I observe that with both versions (3.20.2 vs 3.20.2.2), a JSON response is returned, but the newer version generates an exception, whereas the older version operates without errors. I am attaching Wireshark HTTP/2 stream screenshots to illustrate this behavior.

I noticed that Last-Stream-ID value is 1 in GOAWAY stream ( sent from server ) in both cases, also HEADERS and DATA are sent with stream id 1 in both cases.
Also, these HEADERS and DATA are received after GOAWAY stream while change done in Http2ConnectionBase class adds check to read Headers if GOAWAY is null.

I’m not sure whether my explanation makes sense.

[geoand]

@vietj ^