Duplicated and 0 value opentelemetry metrics messages

[rsvoboda]

Describe the bug

I have a simple application using opentelemetry metrics with opentelemetry-exporter-logging (https://github.com/rsvoboda/code-with-quarkus-otel).
When inspecting the logs for sent messages, I noticed some issues that should be discussed and possibly fixed.

There are 2 main categories:

• sort of duplicated messages
• messages with 0 value attribute

Duplicated messages
2 messages are sent at the "same" time, the only difference seems to be InstrumentationScopeInfo which is io.opentelemetry.runtime-telemetry-java8 or io.opentelemetry.runtime-telemetry-java17
Affected metrics:

• jvm.cpu.recent_utilization
• jvm.class.count
• jvm.class.unloaded
• jvm.class.loaded

I think there should be just messages with one instrumentation scope sent. Motivation is to limit the used bandwidth and CPU cycles on both the sender and receiver sides.

Messages with 0 value attribute
Messages with InstrumentationScopeInfo set to io.opentelemetry.runtime-telemetry-java17 have value attribute set to 0, the ones with io.opentelemetry.runtime-telemetry-java8 provide data
Affected metrics:

• jvm.memory.used_after_last_gc
• jvm.memory.limit
• jvm.memory.committed
• jvm.memory.used

0 value is suspicious, probably something is not initiated well or just the messages with InstrumentationScopeInfo set to io.opentelemetry.runtime-telemetry-java8 should be sent. Motivation is to avoid sending invalid data and possibly limit the used bandwidth.

Examples:
Sent messages are captured in https://github.com/rsvoboda/code-with-quarkus-otel/blob/main/README.md or you can get them using the reproducer steps below.

Additional jvm.thread.count topic
Note about the description difference in sent messages: Number of executing threads vs. Number of executing platform threads. for io.opentelemetry.runtime-telemetry-java17 vs. io.opentelemetry.runtime-telemetry-java8
Also java8 one provides more details, java17 is a summary, but those details are complementary, so imho no adjustments needed on the details provided.

CC @brunobat

How to Reproduce?

Get https://github.com/rsvoboda/code-with-quarkus-otel

Run

ATTRIBUTES=(
jvm.memory.used
jvm.memory.committed
jvm.memory.limit
jvm.memory.used_after_last_gc
jvm.gc.duration
jvm.thread.count
jvm.class.loaded
jvm.class.unloaded
jvm.class.count
jvm.cpu.time
jvm.cpu.count
jvm.cpu.recent_utilization
)

mvn clean package -DskipTests

for ATTRIBUTE in ${ATTRIBUTES[*]}
do
  echo "===========  $ATTRIBUTE  ==========="
  java -jar target/quarkus-app/quarkus-run.jar | grep "$ATTRIBUTE," &
  for i in {1..500}; do curl -s http://localhost:8080/hello > /dev/null ; sleep 0.001; done
  sleep 10
  PID=`jps -l | grep target/quarkus-app/quarkus-run.jar | cut -d" " -f 1`
  kill $PID
  echo ""
done

Output of uname -a or ver

macOS

Output of java -version

Java 21

Quarkus version or git rev

Quarkus main

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @brunobat (opentelemetry), @ebullient (metrics), @jmartisk (metrics), @radcortez (opentelemetry)

[andcip]

Hello, I encountered the same issue.

I was able to reproduce it locally as well, using a simple Quarkus starter project with version 3.20.1.redhat-00003 and Java version openjdk 21.0.2 2024-01-16 LTS.

It's impossible to exclude the opentelemetry-runtime-telemetry-java8 dependency, as it causes compilation issues.
Thanks

[brunobat]

Will look into it.

[brunobat]

These metrics are created using standard OpenTelemetry instrumentation. Please note that opentelemetry-runtime-telemetry-java8 is a dependency of io.opentelemetry.runtime-telemetry-java17.
See: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/4774069ebfa8cad252696079f61113b2d06da59f/instrumentation/runtime-telemetry/runtime-telemetry-java17/library/README.md

The metrics gathered by the two implementations (jdk8 and jdk17) are mutually exclusive and the union of them produces the full set of available metrics.

Will create a PR to minimize overlaps.

[brunobat]

It's impossible to exclude the opentelemetry-runtime-telemetry-java8 dependency, as it causes compilation issues. Thanks

What compilation issues, @andcip ?

[andcip]

Hi @brunobat ,
Thank you very much for the support. Last week, both the client and I were getting a compilation error, but as of today, it doesn't seem to cause any issues. However, the exclusion has no effect.
After a quick analysis, I see that the OTLP Java 8 library is a dependency of the Java 17 one. If we can extract it and set it as a direct dependency of quarkus-opentelemetry, it would be easy to exclude. Otherwise, we need to consider a system that detects the Java version and disables metrics for the unsupported version.

[brunobat]

@andcip please let me explain in a different way.
We will not remove opentelemetry-runtime-telemetry-java8from the dependency tree.

opentelemetry-runtime-telemetry-java8 produces some jvm metrics and io.opentelemetry.runtime-telemetry-java17 produces others. Incidentally, some of them overlap and the work I'm doing will minimize this overlap.

Please note that the library name includes java8 not because it for Java 8 but because it can be used in Java 8+ jvms. Similarly, the java17 lib is for Java 17+ jvms.