micrometer registry otlp: jvm_gc_* and jvm_buffer_* metrics missing for native image

[patpatpat123]

Hello team, hello @jonatan-ivanov ,

Just wanted to reach out with a small issue I am observing.
After investigation, I believe this is related to native image.

Background, I have a springboot app, and with it, actuator and registry otlp.
I am trying to send the metrics to a backend, in this example, grafana cloud, just for the sake of having some visual of the missing metrics.

With the app running in "normal" mode (not a native image), all metrics are present.

Now building the exact same app, as native image, and run the dicker container native image.
It is observed that some metrics are missing.

Here is the minimal reproducible repository https://github.com/patpatpat123/nativejvmmetricsissueminimal
The backend is available at https://kafkaobservability.grafana.net/explore?schemaVersion=1&panes=%7B%22tj0%22%3A%7B%22datasource%22%3A%22grafanacloud-prom%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%22%2C%22range%22%3Atrue%2C%22instant%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22grafanacloud-prom%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-1h%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1

Could you please kindly look at this issue? The missing metrics are quite important ones.

Thank you so much

[jonatan-ivanov]

This is a duplicate of a conversation we had on Slack. Have you checked with your debugger if the Micrometer binders get the data from JMX?
SubstrateVM (the runtime of GraalVM native-imeage) does not support everything on JMX. Upgrading Micrometer and/or Java might also help.

[patpatpat123]

Thanks for taking a look.
I am not sure how to check with debugger if "Micrometer binders get the data from JMX"

If SubstrateVM does not support everything on JMX, how can I reach out to them to ask?

In terms of parity, it seems there is an issue here, a non native app, has the metrics, while a native app does not.

I am already on the latest springboot 3.4, and JDK23. Could you suggest which versions should I use to mitigate the issue?

Minimal updated: https://github.com/patpatpat123/nativejvmmetricsissueminimal

[jonatan-ivanov]

I am not sure how to check with debugger if "Micrometer binders get the data from JMX"

1. Find which binder registers the metric that is missing
2. Place a breakpoint to that binder
3. See what JMX returns to Micrometer
   If you don't know how to debug, please check the docs of your IDE.

If SubstrateVM does not support everything on JMX, how can I reach out to them to ask?

GraalVM GitHub repo I guess.

Could you suggest which versions should I use to mitigate the issue?

If you are already on latest, you don't need to upgrade. :)

[patpatpat123]

Can you also confirm with the minimal (enough?) reproducible the issue is in fact happening?

I reached out the GraalVM internal team, which would like confirmation from members of micrometer.

And if I understand correctly, this is not a micrometer issue. Am I correct?

[jonatan-ivanov]

Can you also confirm with the minimal (enough?) reproducible the issue is in fact happening?

Please check first if the issue is JMX returning data or not.

I reached out the GraalVM internal team, which would like confirmation from members of micrometer.

Please link the issue where the GraalVM team asks this, they can also comment on this issue.

And if I understand correctly, this is not a micrometer issue. Am I correct?

I would say probably not a Micrometer issue but it's hard to tell without you confirming what JMX returns.

[shakuzen]

For the GC metrics, see #5149 and oracle/graal#8237. Until the GraalVM issue is resolved and released, it's expected that GC metrics will be missing in GraalVM native images.

For the buffer metrics, SubstrateVM needs to provide an implementation of BufferPoolMXBean which I believe it currently doesn't. I didn't find any issue in their issue tracker requesting it, so it would be a good idea to open one so they know there is a need for it.

I don't think there's anything we can do about this in Micrometer. Once the features are available in GraalVM, the existing MeterBinders should work without changes in Micrometer. Given that, I'll close this issue. Let us know if we've missed anything.