Requests with larger payload to gRPC server hang when using Vert.x gRPC implementation (use-separate-server=false)

[Ultranium]

Describe the bug

When using the Vert.x gRPC server implementation (use-separate-server=false), requests with larger payload (10MB+) just hang, request handler code on the server side is not executed.
Legacy gRPC implementation (use-separate-server=true) works in the same environment just fine.

Expected behavior

Normal request processing.

Actual behavior

Requests hang, request handler is not called.

How to Reproduce?

Here is a simple reproducer: https://github.com/Ultranium/quarkus-grpc-vertx-reproducer
The unit test file HelloGrpcServiceTest has a test method testHello(), which sends a few 10MB requests to the gRPC server and times out when waiting for replies.

Changing the quarkus.grpc.server.use-separate-server setting in application.properties to true makes this test pass.

Output of uname -a or ver

Linux desktop 6.13.5-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 27 15:07:31 UTC 2025 x86_64 GNU/Linux

Output of java -version

openjdk version "21.0.7" 2025-04-15

Quarkus version or git rev

3.23.4

Build tool (ie. output of mvnw --version or gradlew --version)

Maven 3.9.9

Additional information

No response

[quarkus-bot]

/cc @alesj (grpc), @cescoffier (grpc)

[cescoffier]

@vietj any idea?

[vietj]

@cescoffier I'll have a look at it soon

[vietj]

The timeout is too small, on my laptop this test requires at least 16 seconds to pass.

[Ultranium]

@vietj I increased timeout and indeed the test now passes after 25 seconds on my machine.
But switching to legacy gRPC implementation makes this test complete in about one second, so there is clearly a performance issue here.

I faced this issue on a real project, where switching to the Vertx gRPC server made my service basically unusable due to lots of requests timing out, the larger the payload the higher the risk of a failed request.
Interestingly enough, everything works more or less stable when size of requests is <1MB.

[vietj]

I think our handling of large message is not efficient and we should investigate this.

I suspect it could come from the HTTP/2 window frame size, that might be too small. I doubt it's a CPU issue (it could be)

cc @franz1981

[franz1981]

But switching to legacy gRPC implementation makes this test complete in about one second, so there is clearly a performance issue here.

the "legacy" is still using the Netty grpc handlers on top of vertx quarkus event loops?
see #48526 (comment)

In any case having some allocation and cpu profiling data via async-profiler can tell something useful , but if is as @vietj said, maybe not

[Ultranium]

the "legacy" is still using the Netty grpc handlers on top of vertx quarkus event loops?

I think so (I'm not an expert on Quarkus internals though).
The code in both cases is identical on the server and the client side, the only difference is the quarkus.grpc.server.use-separate-server setting value.