Massive performance drop when retrieving `InputStream` using REST client

[LarsSven]

Describe the bug

We were working on moving some code over from a non-Quarkus REST client to a Quarkus REST client. We noticed a massive drop in performance when we dit that, and got some very interesting results.

When using the Quarkus client to download a file from a URL as an InputStream, the creation of the InputStream object (So not actually streaming the file, but the creation of the InputStream) scales worse than linearly with the size of the file you are downloading. Getting the InputStream however should take a constant amount of time and be milliseconds of work.

We ran a test against a set of object sizes we uploaded to MinIO (and thus the REST client simply sees a download link for the file):

1 * 1024,          // 1KB
1 * 1024 * 1024,   // 1MB
10 * 1024 * 1024,  // 10MB
100 * 1024 * 1024, // 100MB
250 * 1024 * 1024, // 250MB
500 * 1024 * 1024, // 500MB
1L * 1024 * 1024 * 1024 // 1GB

The default REST client produced these results:

Downloading resource_1024_bytes_14840480614219054265.bin using Default REST client
Time to generate InputStream: 27 ms
Time to stream to file: 0 ms
Downloading resource_1048576_bytes_13817578619309500621.bin using Default REST client
Time to generate InputStream: 3 ms
Time to stream to file: 3 ms
Downloading resource_10485760_bytes_2310616242777621351.bin using Default REST client
Time to generate InputStream: 1 ms
Time to stream to file: 22 ms
Downloading resource_104857600_bytes_8607201645369659057.bin using Default REST client
Time to generate InputStream: 2 ms
Time to stream to file: 114 ms
Downloading resource_262144000_bytes_8533175391852326467.bin using Default REST client
Time to generate InputStream: 3 ms
Time to stream to file: 248 ms
Downloading resource_524288000_bytes_16558373151874773264.bin using Default REST client
Time to generate InputStream: 2 ms
Time to stream to file: 346 ms
Downloading resource_1073741824_bytes_14053281376256782222.bin using Default REST client
Time to generate InputStream: 2 ms
Time to stream to file: 686 ms

The Quarkus REST client however produced these results:

Downloading resource_1024_bytes_14840480614219054265.bin using Quarkus REST client
Time to generate InputStream: 68 ms
Time to stream to file: 0 ms
Downloading resource_1048576_bytes_13817578619309500621.bin using Quarkus REST client
Time to generate InputStream: 5 ms
Time to stream to file: 0 ms
Downloading resource_10485760_bytes_2310616242777621351.bin using Quarkus REST client
Time to generate InputStream: 16 ms
Time to stream to file: 3 ms
Downloading resource_104857600_bytes_8607201645369659057.bin using Quarkus REST client
Time to generate InputStream: 234 ms
Time to stream to file: 28 ms
Downloading resource_262144000_bytes_8533175391852326467.bin using Quarkus REST client
Time to generate InputStream: 1123 ms
Time to stream to file: 69 ms
Downloading resource_524288000_bytes_16558373151874773264.bin using Quarkus REST client
Time to generate InputStream: 3837 ms
Time to stream to file: 136 ms
Downloading resource_1073741824_bytes_14053281376256782222.bin using Quarkus REST client
Time to generate InputStream: 15140 ms
Time to stream to file: 268 ms

Expected behavior

Generating the InputStream should take a few milliseconds and stay constant no matter how much data needs to be streamed

Actual behavior

The InputStream generation scales with a time complexity that is worse than linear and takes over 15 seconds to create the InputStream for a 1GB file.

How to Reproduce?

1. Clone the reproducer: https://gitlab.com/l.s.andringa1/quarkus-inputstream-reproducer
2. Run ./gradlew quarkusRun

Output of uname -a or ver

Ubuntu 25.04

Output of java -version

Java 21.0.7

Quarkus version or git rev

3.24.2

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 8.14.2

Additional information

No response

[quarkus-bot]

/cc @cescoffier (rest-client), @geoand (rest-client)

[gsmet]

Can you clarify what you call "non-Quarkus REST client"?

I think we arealdy discussed a similar issue in the past but I can't find it atm.

I draw some graphs showing that we were allocating like crazy when the size of the file grew.

I remember generating this data that demonstrates the issue:

We improved things a bit since then but I'm pretty sure the core of the issue is still there.

[gsmet]

IIRC one of the problems was the use of small chunks as you ended up resizing the buffer a lot, which led to a loooot of allocations.

When uploading in chunks, the receiving side don't have the size of the file and that was the most prominent cause of the issue.

Could you try setting [quarkus.rest-client.max-chunk-size](https://quarkus.io/guides/rest-client#quarkus-rest-client-config_quarkus-rest-client-max-chunk-size) to 1 MB and see if it improve things?

Also it would be interesting to check the difference between how the HTTP requests are handled for the two clients.

[geoand]

I think we arealdy discussed a similar issue in the past but I can't find it atm.

I am pretty sure we have, I'll try and find it tomorrow

[gsmet]

I remember I posted the graph and that Franz and you were involved in the discussion but no idea where it is...

[gsmet]

@LarsSven btw, I don't think you pushed your reproducer. I only see a README there.

[gsmet]

I think the original issue that triggered the discussion was #45122 but then I don't know where the discussion about the massive memory allocations happened.

Now I'm not sure it's the issue here, we definitely need to see the reproducer.

[LarsSven]

My apologies, I have pushed the reproducer to the repository @gsmet @geoand

[geoand]

I think this comment applies here as did in #47884, but I'll have to check the reproducer when I have some time.

[geoand]

By using Response as the return type, you are forcing the client to buffer everything in memory

[LarsSven]

For our production code we need the content-length though, which I am getting from the response object.

[LarsSven]

Basically this is what we are trying to do in our production code:

fun downloadInputResource(
    inputResource: RemoteInputResource
): LocalInputResource {
    val response = resourceRestClient.downloadResource(inputResource.downloadUrl.toString())

    if (response.status != 200) {
        val errorBody = response.readEntity(String::class.java)
        throw Exception("Failed to download resource: ${response.status} - $errorBody")
    }

    val contentStream = response.entity as InputStream
    val contentLength = response.getHeaderString("Content-Length")?.toLong()
        ?: throw IllegalStateException("Missing Content-Length header")

    return LocalInputResource(
        uri = inputResource.uri,
        contentStream = contentStream,
        contentLength = contentLength,
        metadata = inputResource.metadata
    )
}

https://gitlab.com/rug-digitallab/products/themis/runtimes/kotlin-runtimes/-/blob/ls/197-resource-urls/common-runtime/runtime-base/src/main/kotlin/nl/rug/digitallab/themis/runtime/common/base/runner/resources/ResourceManager.kt?ref_type=heads

[gsmet]

By using Response as the return type, you are forcing the client to buffer everything in memory

Sorry for being candid but why are we buffering everything in memory? We know this won't scale and in several occasions, it screwed some of our users.

Could we create temp files instead? Files are getting larger and us not being able to handle this properly isn't exactly nice.

Now again, this is a very candid question :).