Updating from quarkus 2.9.2.Final (Java 17) to 3.17.2 (Java 21) heap memory issues

[TiagoFerreirahub]

Describe the bug

After the following changes:

• Changing quarkus version from 2.9.2.Final to 3.17.2;
• Changing dependency "quarkus-resteasy" to "quarkus-rest-jackson"

A rest client that was consuming a ~350MB json response stopped working and now throws java.lang.OutOfMemoryError: Java heap space.

Same heap memory configurations were used:

• Xms100m
• Xmx300m

In order for the new quarkus version and dependencies to work i needed to raise the Xmx to 2g.

I was not able to upload all the zip files, but i do have the whole setup for testing done, if needed i can send it.

Expected behavior

Client-works.zip
In version 3.17.2 of quarkus, with the new dependencies and java version, the json body should be able to be consumed with similar heap configurations. (Working client with quarkus version 2.9.2.Final on zip file in attachments)

Actual behavior

After changing the quarkus version to 3.17.2, java to version 21, and changing dependency "quarkus-resteasy" to "quarkus-rest-jackson", it is not possible to consume the same json body response with the same heap memory configurations. (Needed to raise Xmx from 300m to 2g)

How to Reproduce?

1. Create a quarkus project with version 2.9.2.Final, java 17 and "quarkus-resteasy" and "quarkus-rest-client-jackson" dependencies. (Check zip file "Client-works.zip") and run it with Xms100m and Xmx300m.
2. Create another quarkus client similar to the previous one but with version 3.17.2, java 21 and with "quarkus-rest" and "quarkus-rest-client-jackson" dependencies and run it with Xms100m and Xmx300m.
3. Create a mock server that will return a ~350MB json response.
4. Make a request to the server on one client and the other, the client with quarkus version 2.9.2.Final should be able to read the response and the client with the quarkus version 3.17.2 should take a lot of time to read it and eventually throw java.lang.OutOfMemoryError: Java heap space.

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

• I ran all the tests on Docker containers with the specified heap configurations.

[geoand]

Thanks a lot for reporting.

Changing dependency "quarkus-resteasy" to "quarkus-rest-jackson"

This is an equivalent change. The apples to apples comparison would be from quarkus-resteasy to quarkus-rest

After changing the quarkus version to 3.17.2, java to version 21, and changing dependency "quarkus-resteasy" to "quarkus-rest-jackson", it is not possible to consume the same json body response with the same heap memory configurations

This is a lot of changes. I propose doing changes in isolated steps and seeing what the impact is for each one

[quarkus-bot]

/cc @FroMage (rest), @stuartwdouglas (rest)

[TiagoFerreirahub]

I got a chance to perform some isolated changes and :

• Java version does not matter, i tested quarkus 3.17.2 with Java 17 and 21, both gave me the OOM issue described here.
• I also tested using quarkus-rest instead of quarkus-rest-jackson to be as similar as possible, and it also did not help.
• The only thing that seems to have an impact is the quarkus version. (2.9.2.Final works and 3.17.2 does not when processing a ~350MB json response

Maybe im missing something here, but i'm out of ideas right now for more tests that i could do on my side

[gsmet]

The next step is to get a heap dump from the OOM and have a look at it with either VisualVM or Eclipse MAT.

Also bisecting the version might be useful as it's a huuuuge gap. I would try 2.16 then 3.0, then 3.8, then 3.15 and refine depending on the results.

I can help but setting up things can be a bit time consuming and I wouldn't like to shoot in the dark. So if you could provide an easy way to reproduce it with all pieces available, that would already be awesome.

I see you have it ready so if you can send me a link at <github handle>@redhat.com, I could have a closer look.

[geoand]

then 3.0, then 3.8,

I would add 3.2 here also

[TiagoFerreirahub]

Hi there, i looked into the heap dump of the real app where this issue appeared first but i'm not very proficient in looking into heap dumps so i could not pinpoint the issue, would you be able to have a look into it?

@gsmet i sent you an email with both a mock server that generates the response and the client that asks for it that i'm using to test these scenarios.

I tested with java 17 only migrating from quarkus 2.9.2.Final -> 2.16 -> 3.0 -> 3.2 -> 3.8 and when i got to 3.15 it broke (after i changed dependency from quarkus-resteasy to quarkus-rest)

If you think it is necessary i can send you a heap dump, but i think you can generate that easily with the projects @gsmet.

Tell me if i can perform any more tests to take out some other options, i'm open to suggestions, and thank you for your help

[TiagoFerreirahub]

Were you able to replicate it @gsmet? Is there any news on this topic?

[gsmet]

Sorry, I completely missed the ping (and the email...). I'll try to have a look later today once I'm done with my current task!

[gsmet]

I'll send you a follow up by email.