IllegalStateException: GlobalOpenTelemetry.set has already been called

[sithmein]

Describe the bug

This is the same as #22805 but with the most recent LTS and a reproducer.
In short, when you access an OpenTelemetry Tracer during a StartupEvent, Quarkus has not initialized OpenTelemetry yet. This leads to a no-op OpenTelemetry instance being set as the singleton instance. Once OpenTelemetry is access by some other means configuration happens (again) and setting the global singleton will fail.
Here is a concrete error. Not that the last cause is the first call that set the singleton instance for the first time.

io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [Le6zQbzkojAYO_OiKIQWJf4lGa4]: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.doCreate(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
	at io.quarkus.opentelemetry.runtime.tracing.intrumentation.restclient.OpenTelemetryClientFilter_Bean.doCreate(Unknown Source)
	at io.quarkus.opentelemetry.runtime.tracing.intrumentation.restclient.OpenTelemetryClientFilter_Bean.create(Unknown Source)
	at io.quarkus.opentelemetry.runtime.tracing.intrumentation.restclient.OpenTelemetryClientFilter_Bean.get(Unknown Source)
	at io.quarkus.opentelemetry.runtime.tracing.intrumentation.restclient.OpenTelemetryClientFilter_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:570)
	at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:550)
	at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:583)
	at io.quarkus.arc.impl.ArcContainerImpl$3.get(ArcContainerImpl.java:337)
	at io.quarkus.arc.impl.ArcContainerImpl$3.get(ArcContainerImpl.java:334)
	at io.quarkus.resteasy.common.runtime.QuarkusConstructorInjector.construct(QuarkusConstructorInjector.java:39)
	at org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.injectedInstance(ResteasyProviderFactoryImpl.java:1314)
	at org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl$AbstractInterceptorFactory.createInterceptor(JaxrsInterceptorRegistryImpl.java:132)
	at org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl$OnDemandInterceptorFactory.initialize(JaxrsInterceptorRegistryImpl.java:146)
	at org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl$OnDemandInterceptorFactory.checkInitialize(JaxrsInterceptorRegistryImpl.java:157)
	at org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl$OnDemandInterceptorFactory.getInterceptor(JaxrsInterceptorRegistryImpl.java:165)
	at org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl$AbstractInterceptorFactory.postMatch(JaxrsInterceptorRegistryImpl.java:122)
	at org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl.postMatch(JaxrsInterceptorRegistryImpl.java:246)
	at org.jboss.resteasy.client.jaxrs.internal.ClientConfiguration.getRequestFilters(ClientConfiguration.java:111)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.getRequestFilters(ClientInvocation.java:399)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:639)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:424)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:71)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler.hasSessionAttached(JobLifecycleHandler.java:321)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler.swapJobs(JobLifecycleHandler.java:240)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler.catcher(JobLifecycleHandler.java:181)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler.lambda$onStart$0(JobLifecycleHandler.java:151)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:527)
	at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:74)
	at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:60)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.createSynthetic(Unknown Source)
	... 44 more
Caused by: java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.
	at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:107)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.maybeSetAsGlobal(AutoConfiguredOpenTelemetrySdkBuilder.java:594)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:508)
	... 47 more
Caused by: java.lang.Throwable
	at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:115)
	at io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:85)
	at io.opentelemetry.api.GlobalOpenTelemetry.getTracer(GlobalOpenTelemetry.java:134)
	at io.quarkus.opentelemetry.runtime.tracing.cdi.TracerProducer.getTracer(TracerProducer.java:33)
	at io.quarkus.opentelemetry.runtime.tracing.cdi.TracerProducer_ProducerMethod_getTracer_2WbgLITcyJ8m3eLEKU0XTuO1Xs4_Bean.doCreate(Unknown Source)
	at io.quarkus.opentelemetry.runtime.tracing.cdi.TracerProducer_ProducerMethod_getTracer_2WbgLITcyJ8m3eLEKU0XTuO1Xs4_Bean.create(Unknown Source)
	at io.quarkus.opentelemetry.runtime.tracing.cdi.TracerProducer_ProducerMethod_getTracer_2WbgLITcyJ8m3eLEKU0XTuO1Xs4_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c33(Unknown Source)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:23)
	at io.opentelemetry.api.trace.TracerProducer_ProducerMethod_getTracer_2WbgLITcyJ8m3eLEKU0XTuO1Xs4_ClientProxy.arc$delegate(Unknown Source)
	at io.opentelemetry.api.trace.TracerProducer_ProducerMethod_getTracer_2WbgLITcyJ8m3eLEKU0XTuO1Xs4_ClientProxy.spanBuilder(Unknown Source)
	at com.knime.enterprise.cloud.services.shared.jpa.otel.CommitSpanner.beforeCompletion(CommitSpanner.java:51)
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:52)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:348)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:66)
	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:135)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1307)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:104)
	at io.quarkus.narayana.jta.runtime.NotifyingTransactionManager.commit(NotifyingTransactionManager.java:70)
	at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.endTransaction(TransactionalInterceptorBase.java:406)
	at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:171)
	at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:107)
	at com.knime.enterprise.cloud.services.shared.jpa.TransactionalRetryInterceptor.doIntercept(TransactionalRetryInterceptor.java:83)
	at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:61)
	at com.knime.enterprise.cloud.services.shared.jpa.TransactionalRetryInterceptor.intercept(TransactionalRetryInterceptor.java:70)
	at com.knime.enterprise.cloud.services.shared.jpa.TransactionalRetryInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler_Subclass.getPurgeAbleJobIdsFromDB(Unknown Source)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler.purgeDiscardedJobs(JobLifecycleHandler.java:630)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler.catcher(JobLifecycleHandler.java:181)
	at com.knime.enterprise.cloud.services.execution.lifecycle.JobLifecycleHandler.lambda$onStart$3(JobLifecycleHandler.java:157)
	... 6 more

Expected behavior

OpenTelemetry should be configured once through the same mechanism regardless how it is called.

Actual behavior

There are at least two initialization paths for OpenTelemetry which are incompatible with each other and lead to an exception.

How to Reproduce?

A reproducer can be found at https://github.com/sithmein/bugs-in-quarkus, specifically quarkus.bugs.TracerInitBean and quarkus.bugs.OpenTelemetryInitTest.

Output of uname -a or ver

Linux crest 6.15.9-201.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Aug 2 11:37:34 UTC 2025 x86_64 GNU/Linux

Output of java -version

21.0.8

Quarkus version or git rev

3.20.2

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.11

Additional information

No response

[quarkus-bot]

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

[brunobat]

@mkouba, shouldn't the StartupEvent be triggered only after Quarkus has finished starting up?

[mkouba]

@mkouba, shouldn't the StartupEvent be triggered only after Quarkus has finished starting up?

Define "after Quarkus has finished starting up" 🤷 . It's currently fired when the CDI container is ready and after all recorder invocations from build steps that also produce a ServiceStartBuildItem complete.

[brunobat]

That's the thing, The OpenTelemetry instance is created through a synthetic bean and it doesn't seem to exist by the time the StartupEvent is fired.

[mkouba]

That's the thing, The OpenTelemetry instance is created through a synthetic bean and it doesn't seem to exist by the time the StartupEvent is fired.

From the stack trace I can see that "GlobalOpenTelemetry.set has already been called" while creating the bean instance of the OpenTelemetry synthetic bean (which is triggered on the first use of the bean). Not sure what you mean with "it doesn't seem to exist by the time the StartupEvent is fired"?

[brunobat]

Correct, This is because OpenTelemetry was used, most likely by the rest client and the default instance was created by the OTel SDK.
This is an known issue and Tracing must not be used before Quarkus is completely up, which it seems is not the case when StartupEvent is issued.
Is there other lifecycle event fired when all is up?... Should we create one?

[mkouba]

This is an known issue and Tracing must not be used before Quarkus is completely up

Well, we should first specify what "Quarkus is completely up" actually means. AFAIK we only have StartupEvent and CDI events for HTTP server start.

[brunobat]

Probably the best time would be when we write in the console that Quarkus started in x seconds.
But it's complex. I've seen ppl using rest clients to populate data on startup event that is needed before the app is ready, which might also affect readiness (Healthcheck).
Right now, the rest client provided by the extension should not be used on startup event, at least if observability is needed. Users should create a rest client from builder and use it.
Ideally, the startup event should only be triggered when all the services in the server are available which is not the case today.