Native Build fails with io.confluent:kafka-streams-avro-serde to v7.8.0

[teimyBr]

Describe the bug

Native Build fails with io.confluent:kafka-streams-avro-serde to v7.8.0

With newer Version then 7.5.1 Quarkus Native Build fails.

Error: An object of type 'org.apache.avro.reflect.ReflectData$AllowNull' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.

Expected behavior

Native Build works with every io.confluent:kafka-streams-avro-serde version

Actual behavior

Qurkus Native Build show this Error
Error: An object of type 'org.apache.avro.reflect.ReflectData$AllowNull' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.

How to Reproduce?

1. Start buildNative with io.confluent:kafka-streams-avro-serde greater then 7.5.1
2. Build fails

Output of uname -a or ver

No response

Output of java -version

Java version: 21.0.2+13

Mandrel or GraalVM version (if different from Java)

GraalVM CE 21.0.2+13.1

Quarkus version or git rev

3.17.7

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 8.12

Additional information

No response

[quarkus-bot]

/cc @Karm (native-image), @alesj (kafka,kafka-streams), @cescoffier (kafka), @galderz (native-image), @gunnarmorling (kafka-streams), @ozangunalp (kafka,kafka-streams), @rquinio (kafka-streams), @zakkak (native-image)

[cescoffier]

Did you check https://quarkus.io/guides/kafka-schema-registry-avro#confluent?

[teimyBr]

In JVM mode, any version of io.confluent:kafka-avro-serializer can be used. In native mode, Quarkus supports the following versions: 6.2.x, 7.0.x, 7.1.x, 7.2.x, 7.3.x.

For version 7.4.x and 7.5.x, due to an issue with the Confluent Schema Serializer, you need to add another dependency

7.5.x is quite old.

I guess Quarkus should not depend on such old versions. I guess everyone want to use quite new versions

[cescoffier]

We should rewrite with something like: for any version > 7.4...

[teimyBr]

io.confluent:kafka-streams-avro-serde to v7.5.1 works quite well on your side, but i would be cool if we can also use 7.6.x, 7.7.x, or even 7.8.x

7.4.x and 7.5.x is quite old

[ozangunalp]

Quarkus doesn't depend on confluent serde artifacts, in any way, they are published to Confluent maven repository not to the Maven central. So yes we should write something like "> 7.4".

[teimyBr]

If i want to use implementation 'io.confluent:kafka-streams-avro-serde:7.8.0' how i can get the Native Build running without failing ?

[cescoffier]

You may need extra native compilation metadata. I would recommend asking directly to Confluent. They can add the metadata in the jar directly.

Do you have a stack trace or something?

[teimyBr]

|| #17 109.7 > Task :quarkusAppPartsBuild
|| #17 111.3 
|| #17 111.3 [1/8] Initializing...                                                                                   (18.0s @ 0.18GB)
|| #17 111.3  Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
|| #17 111.3  Graal compiler: optimization level: 2, target machine: x86-64-v3
|| #17 111.3  C compiler: gcc (redhat, x86_64, 11.4.1)
|| #17 111.3  Garbage collector: Serial GC (max heap size: 80% of RAM)
|| #17 111.3  4 user-specific feature(s):
|| #17 111.3  - com.oracle.svm.thirdparty.gson.GsonFeature
|| #17 111.3  - io.quarkus.kafka.streams.runtime.graal.KafkaStreamsFeature
|| #17 111.3  - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
|| #17 111.3  - io.quarkus.runtime.graal.DisableLoggingFeature: Adapts logging during the analysis phase
|| #17 111.3 ------------------------------------------------------------------------------------------------------------------------
|| #17 111.3  5 experimental option(s) unlocked:
|| #17 111.3  - '-H:+AllowFoldMethods' (origin(s): command line)
|| #17 111.3  - '-H:BuildOutputJSONFile' (origin(s): command line)
|| #17 111.3  - '-H:+UseServiceLoaderFeature' (origin(s): command line)
|| #17 111.3  - '-H:AdditionalSecurityProviders' (origin(s): command line)
|| #17 111.3  - '-H:+GenerateBuildArtifactsFile' (origin(s): command line)
|| #17 111.3 ------------------------------------------------------------------------------------------------------------------------
|| #17 111.3 Build resources:
|| #17 111.3  - 5.33GB of memory (78.8% of 6.77GB system memory, set via '-Xmx6G')
|| #17 111.3  - 2 thread(s) (100.0% of 2 available processor(s), determined at start)
|| #17 304.4 [2/8] Performing analysis...  [*****]                                                                  (192.7s @ 2.33GB)
|| #17 304.4    22,705 reachable types   (90.2% of   25,169 total)
|| #17 304.4    34,901 reachable fields  (64.2% of   54,375 total)
|| #17 304.5   115,909 reachable methods (60.2% of  192,499 total)
|| #17 304.5     6,332 types,   260 fields, and 5,472 methods registered for reflection
|| #17 304.5        66 types,    73 fields, and    60 methods registered for JNI access
|| #17 304.5         4 native libraries: dl, pthread, rt, z
|| #17 304.5 
|| #17 304.6 Error: Unsupported features in 2 methods
|| #17 304.6 Detailed message:
|| #17 304.6 Error: An object of type 'org.apache.avro.reflect.ReflectData$AllowNull' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.
|| #17 304.6 This is not allowed for correctness reasons: All objects that are stored in the image heap must be initialized at build time.
|| #17 304.6 
|| #17 304.6 You now have two options to resolve this:
|| #17 304.6 
|| #17 304.6 1) If it is intended that objects of type 'org.apache.avro.reflect.ReflectData$AllowNull' are persisted in the image heap, add 
|| #17 304.6 
|| #17 304.6     '--initialize-at-build-time=org.apache.avro.reflect.ReflectData$AllowNull'
|| #17 304.6 
|| #17 304.6 to the native-image arguments. Note that initializing new types can store additional objects to the heap. It is advised to check the static fields of 'org.apache.avro.reflect.ReflectData$AllowNull' to see if they are safe for build-time initialization,  and that they do not contain any sensitive data that should not become part of the image.
|| #17 304.6 
|| #17 304.6 2) If these objects should not be stored in the image heap, you can use 
|| #17 304.6 
|| #17 304.6     '--trace-object-instantiation=org.apache.avro.reflect.ReflectData$AllowNull'
|| #17 304.6 
|| #17 304.6 to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 
|| #17 304.6 
|| #17 304.6     '--initialize-at-run-time=<culprit>'
|| #17 304.6 
|| #17 304.6 to prevent the instantiation of the object.
|| #17 304.6 
|| #17 304.6 If you are seeing this message after enabling '--strict-image-heap', this means that some objects ended up in the image heap without their type being marked with --initialize-at-build-time.
|| #17 304.6 To fix this, include '--initialize-at-build-time=org.apache.avro.reflect.ReflectData$AllowNull' in your configuration. If the classes do not originate from your code, it is advised to update all library or framework dependencies to the latest version before addressing this error.
|| #17 304.6 Please address this problem to be prepared for future releases of GraalVM.
|| #17 304.6 
|| #17 304.6 The following detailed trace displays from which field in the code the object was reached.
|| #17 304.6 Trace: Object was reached by
|| #17 304.6   trying to constant fold static field io.confluent.kafka.schemaregistry.avro.AvroSchemaUtils.REFLECT_DATA_ALLOW_NULL_INSTANCE
|| #17 304.6     at io.confluent.kafka.schemaregistry.avro.AvroSchemaUtils.getReflectDataAllowNull(AvroSchemaUtils.java:99)
|| #17 304.6   parsing method io.confluent.kafka.schemaregistry.avro.AvroSchemaUtils.getReflectDataAllowNull(AvroSchemaUtils.java:99) reachable via the parsing context
|| #17 304.6     at static root method.(Unknown Source)
|| #17 304.6 
|| #17 304.6 Error: An object of type 'org.apache.avro.reflect.ReflectData' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.
|| #17 304.6 This is not allowed for correctness reasons: All objects that are stored in the image heap must be initialized at build time.

[cescoffier]

Slightly annoying, but I think io.confluent.kafka.schemaregistry.avro.AvroSchemaUtil needs to be runtime initialized now.

[geoand]

Is this still an issue?