Startup-Error: IllegalStateException: GlobalOpenTelemetry.set has already been called. #22805
Description
Describe the bug
Sometimes (5 % of Pipelines) we build Docker-Images that fail to start due to an initialization-error in OpenTelemetry:
Caused by: java.lang.RuntimeException: Failed to start quarkus
at io.quarkus.runner.ApplicationImpl.<clinit>(Unknown Source)
... 15 more
Caused by: java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.
at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:84)
at io.opentelemetry.sdk.OpenTelemetrySdkBuilder.buildAndRegisterGlobal(OpenTelemetrySdkBuilder.java:63)
at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder.createOpenTelemetry(OpenTelemetryRecorder.java:33)
at io.quarkus.deployment.steps.OpenTelemetryProcessor$createOpenTelemetry1944204174.deploy_0(Unknown Source)
at io.quarkus.deployment.steps.OpenTelemetryProcessor$createOpenTelemetry1944204174.deploy(Unknown Source)
... 16 more
Caused by: java.lang.Throwable
at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:92)
at io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:62)
at io.opentelemetry.api.GlobalOpenTelemetry.getTracer(GlobalOpenTelemetry.java:111)
at io.quarkus.opentelemetry.runtime.tracing.TracerProducer.getTracer(TracerProducer.java:2
We build a jar-file using gradle jar quarkusBuild and then add this to a Docker-image. If the error occours, it occours every time the image is used (on developer machine, in the cloud, ... doesn´t matter). If we re-run the pipeline with nearly no code-change (we added a commit with an extra space in a readme) chances are that the error disappears.
What might be application-specific is a Tracing-Interceptor we use. That gets the Tracer injected:
@Provider
@Priority(Interceptor.Priority.APPLICATION)
class TracingInterceptor : ContainerResponseFilter {
override fun filter(ctx: ContainerRequestContext, responseContext: ContainerResponseContext) {
val spanName = ctx.method + " " + ctx.uriInfo.path
val span = Span.current() ?: return
span.updateName(spanName)
val traceId = span.spanContext?.traceId
MDC.put("traceId", traceId)
responseContext.headers.putSingle("TRACE-ID", traceId ?: "no tracing")
}
}
@Interceptor
@Tracing
@Suppress("TooGenericExceptionCaught")
@Priority(Interceptor.Priority.APPLICATION)
class TracingMethodInterceptor {
@Inject
lateinit var tracer: Tracer
@Inject
lateinit var request: RequestAttributes
@ConfigProperty(name = "tracing.logParameters")
var logParameters: Boolean = true
@AroundInvoke
@Throws(Exception::class)
fun trace(ctx: InvocationContext): Any? {
val parentSpan = Span.current()
request.attributes["traceId"] = parentSpan.spanContext.traceId
val methodDisplay = "${ctx.method.name} ${ctx.method.declaringClass.simpleName}"
val fullMethodName = ctx.method.toGenericString()
val span = tracer.spanBuilder(methodDisplay)
.setParent(Context.current().with(parentSpan))
.startSpan()
span.makeCurrent().use {
span.setAttribute("method", fullMethodName)
if (logParameters) {
val parameterDisplay = ctx.parameters.filterNotNull().joinToString { "$it" }
span.setAttribute("parameters", parameterDisplay)
}
try {
return ctx.proceed()
} catch (e: Exception) {
e.logToTracer()
throw e
} finally {
span.end()
}
}
}
}By now we consider this to be a build-problem, since the error is tight to a specific artifact and not to the code or the runtime-environment.
What might be of interest:
- we use kotlin
- we use gradle
- we do not use graalvm
Expected behavior
Every build of the same codebase should deliver reproduceable results.
Actual behavior
On a random basis the error described above occours. We do not have any hint why this happens.
How to Reproduce?
Unfortunately we do not understand the error enough to provide a reproduction-instruction. Maybe you have an idea what might be the problem from the information we provided or you have a hint where we could dig deeper.
Output of uname -a or ver
Linux runner-t9jgdck6-project-9010693-concurrent-0 5.11.0-1026-gcp #29~20.04.1-Ubuntu SMP Fri Jan 7 12:24:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Output of java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment Temurin-11.0.13+8 (build 11.0.13+8)
OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (build 11.0.13+8, mixed mode)
GraalVM version (if different from Java)
No response
Quarkus version or git rev
io.quarkus:quarkus-bom:2.6.1.Final
Build tool (ie. output of mvnw --version or gradlew --version)
------------------------------------------------------------
Gradle 7.3.3
------------------------------------------------------------
Build time: 2021-12-22 12:37:54 UTC
Revision: 6f556c80f945dc54b50e0be633da6c62dbe8dc71
Kotlin: 1.5.31
Groovy: 3.0.9
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 11.0.13 (Eclipse Adoptium 11.0.13+8)
OS: Linux 5.11.0-1026-gcp amd64
Additional information
The Environment-Infos describe the Build-Environment.