Smallrye fault tolerance retry policy goes in StackOverflowError

[joystuffs]

Describe the bug

Hi,

I noticed an unexpected behavior after updating Smallrye fault tolerance

from

 io.quarkus:quarkus-smallrye-fault-tolerance:3.15.3
 io.smallrye:smallrye-fault-tolerance:6.4.0

to

io.quarkus:quarkus-smallrye-fault-tolerance:3.20.0
io.smallrye:smallrye-fault-tolerance:6.9.0

Out of nowhere, I started getting a StackOverflowError, something that never happened in the previous version.

  
java.util.concurrent.CompletionException: java.lang.StackOverflowError at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) at
java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(Unknown Source) at
java.base/java.util.concurrent.CompletableFuture.whenComplete(Unknown Source) 	at
java.base/java.util.concurrent.CompletableFuture$MinimalStage.whenComplete(Unknown Source) at
io.quarkus.scheduler.common.runtime.DefaultInvoker.invoke(DefaultInvoker.java:25) at
io.quarkus.scheduler.common.runtime.DelegateInvoker.invokeDelegate(DelegateInvoker.java:29) at
io.quarkus.scheduler.common.runtime.StatusEmitterInvoker.invoke(StatusEmitterInvoker.java:35) at
io.quarkus.scheduler.common.runtime.DelegateInvoker.invokeDelegate(DelegateInvoker.java:29) at
io.quarkus.scheduler.common.runtime.DelegateInvoker.invokeComplete(DelegateInvoker.java:36) at
io.quarkus.scheduler.common.runtime.OffloadingInvoker$2.call(OffloadingInvoker.java:54) at
io.quarkus.scheduler.common.runtime.OffloadingInvoker$2.call(OffloadingInvoker.java:51) at
io.vertx.core.impl.ContextImpl.lambda$executeBlocking$4(ContextImpl.java:192) at
io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270) at
io.vertx.core.impl.ContextImpl$1.execute(ContextImpl.java:221) at
io.vertx.core.impl.WorkerTask.run(WorkerTask.java:56) at
io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:643) at
org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675) at
org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654) at
org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627) at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594) at
org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11) at
org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11) at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at
java.base/java.lang.Thread.run(Unknown Source)
  

below the programmatically TypedGuard used for the retries.

I followed the official update guide to rewrite the retry policy, even though the actual code change was minimal.

ref : https://smallrye.io/docs/smallrye-fault-tolerance/6.9.0/reference/programmatic-api.html

TypedGuard.create(Response.class)
                .withDescription("retry policy")
                .withRetry()
                .maxDuration(30, ChronoUnit.SECONDS)
                .maxRetries(-1)
                .abortOn(DomainException.class)
                .retryOn(List.of(DomainRetryableException.class))
                .done()
                .withTimeout()
                .duration(10, ChronoUnit.SECONDS)
                .done()
                .build();

maybe there is a new config , I'm missing.

any clue of the stackOverFlow?

Expected behavior

have the same behavior as in the previous version

Actual behavior

Out of nowhere, I started getting a StackOverflowError, something that never happened in the previous version.

How to Reproduce?

below the TypedGuard used for the retry

TypedGuard.create(Response.class)
                .withDescription("retry policy")
                .withRetry()
                .maxDuration(30, ChronoUnit.SECONDS)
                .maxRetries(-1)
                .abortOn(DomainException.class)
                .retryOn(List.of(DomainRetryableException.class))
                .done()
                .withTimeout()
                .duration(10, ChronoUnit.SECONDS)
                .done()
                .build();

Output of uname -a or ver

No response

Output of java -version

21

Quarkus version or git rev

3.20.0

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @Ladicek (fault-tolerance,smallrye), @jmartisk (smallrye), @phillip-kruger (smallrye), @radcortez (smallrye)

[Ladicek]

That shouldn't be occuring, no. There's likely a bug. I'll try to reproduce.

[joystuffs]

@Ladicek I didn't include the entire flow, but basically, there's a scheduler that triggers an event which performs a simple HTTP request, this HTTP request is wrapped with the retry policy.
(you could simulate it using WireMock and return always 4xx to enable the retries).

That said, I don't think the scheduler is relevant. the same bug occurs in other parts of the application that use the same retry policy.

[joystuffs]

I did more tests and below the recurrent exception :

 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-7) Uncaught exception received by Vert.x: java.lang.NoClassDefFoundError: Could not initialize class io.smallrye.faulttolerance.core.retry.RetryEvents$Finished
	at io.smallrye.faulttolerance.core.retry.Retry.afterDelay(Retry.java:160)
	at io.smallrye.faulttolerance.core.retry.Retry.doRetry(Retry.java:69)
	at io.smallrye.faulttolerance.core.retry.Retry.apply(Retry.java:59)
	at io.smallrye.faulttolerance.core.fallback.Fallback.apply(Fallback.java:35)
	at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
	at io.smallrye.faulttolerance.core.metrics.DelegatingMetricsCollector.apply(DelegatingMetricsCollector.java:37)
	at io.smallrye.faulttolerance.core.async.SyncAsyncSplit.apply(SyncAsyncSplit.java:23)
	at io.smallrye.faulttolerance.apiimpl.GuardCommon.guard(GuardCommon.java:69)
	at io.smallrye.faulttolerance.apiimpl.TypedGuardImpl.guard(TypedGuardImpl.java:111)
	at io.smallrye.faulttolerance.apiimpl.TypedGuardImpl.call(TypedGuardImpl.java:97)
	at io.smallrye.faulttolerance.apiimpl.LazyTypedGuard.call(LazyTypedGuard.java:23)
	at example.com.api.Service.runService.process(runService.java:115)
	at example.com.api.Service.runService.lambda$execute$0(runService.java:71)
	at java.base/java.util.HashMap.forEach(HashMap.java:1429)
	at example.com.api.Service.runService(Service.java:71)
	at io.quarkus.vertx.runtime.EventConsumerInvoker.invokeBean(EventConsumerInvoker.java:79)
	at io.quarkus.vertx.runtime.EventConsumerInvoker.invoke(EventConsumerInvoker.java:51)
	at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder$3$1$2.call(VertxEventBusConsumerRecorder.java:150)
	at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder$3$1$2.call(VertxEventBusConsumerRecorder.java:146)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$4(ContextImpl.java:192)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
	at io.vertx.core.impl.ContextImpl$1.execute(ContextImpl.java:221)
	at io.vertx.core.impl.WorkerTask.run(WorkerTask.java:56)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1591)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.StackOverflowError [in thread "vert.x-worker-thread-3"]
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:524)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:517)
	at io.smallrye.faulttolerance.core.retry.RetryEvents$Finished.(RetryEvents.java:18)
	at io.smallrye.faulttolerance.core.retry.Retry.afterDelay(Retry.java:160)
	at io.smallrye.faulttolerance.core.retry.Retry.lambda$doRetry$0(Retry.java:87)
	at io.smallrye.faulttolerance.core.retry.SyncDelayAsAsync.after(SyncDelayAsAsync.java:27)
	at io.smallrye.faulttolerance.core.retry.Retry.doRetry(Retry.java:86)
	at io.smallrye.faulttolerance.core.retry.Retry.lambda$afterDelay$1(Retry.java:154)
	at io.smallrye.faulttolerance.core.FutureImpl.attemptDelivery(FutureImpl.java:159)
	at io.smallrye.faulttolerance.core.FutureImpl.then(FutureImpl.java:90)
	at io.smallrye.faulttolerance.core.retry.Retry.afterDelay(Retry.java:129)
	at io.smallrye.faulttolerance.core.retry.Retry.lambda$doRetry$0(Retry.java:87)
	at io.smallrye.faulttolerance.core.retry.SyncDelayAsAsync.after(SyncDelayAsAsync.java:27)
	at io.smallrye.faulttolerance.core.retry.Retry.doRetry(Retry.java:86)...
 

The stack trace goes on indefinitely, suggesting repeated retry attempts.

this doesn't happen at all with

io.quarkus:quarkus-smallrye-fault-tolerance:3.15.3
io.smallrye:smallrye-fault-tolerance:6.4.0

[Ladicek]

I guess I know why this happens, the synchronous retry implementation used a while loop, but it uses recursion now (after the big async unification). We'll probably have to add a trampoline, or something like that.

[joystuffs]

Good to know, thanks for the feedback!

Looking at the SmallRye blog, I noticed there was a major refactoring starting with version 6.7.0:

https://smallrye.io/blog/fault-tolerance-6-7-0/

I'm not sure if the issue you mentioned is related to that.

assuming I want to keep Quarkus 3.20, what would you suggest in this case?

[Ladicek]

That is indeed the cause. I should be able to fix the issue next week or the week after that, and that fix should be backportable to 3.20.

[Ladicek]

This should be fixed by smallrye/smallrye-fault-tolerance#1120

[Ladicek]

Gee, I didn't mean to close this by merging the upstream PR. This issue is still valid, should be closed by bumping SRye FT to a new version.