Invalid request number, must be greater than 0 when sending message to outbound channel #48624
Description
Describe the bug
Hope that's the right place to ask guidance on this.
We are currently making use of the quarkus-messaging-amqp extension and we have set-up some outbound channels where we are pushing messages on some conditions.
Sometimes we hit the following error Invalid request number, must be greater than 0, here the full stacktrace:
2025-06-25 10:59:45,580 645a1466d4ef /usr/lib/jvm/java-17-openjdk-17.0.15.0.6-2.el9.x86_64/bin/java[7] ERROR [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-1) SRMSG16225: Failure reported for channel `dataset-event-out`, closing client: java.lang.IllegalArgumentException: Invalid request number, must be greater than 0
at io.smallrye.mutiny.helpers.Subscriptions.getInvalidRequestException(Subscriptions.java:24)
at io.smallrye.mutiny.operators.multi.MultiOperatorProcessor.request(MultiOperatorProcessor.java:117)
at io.smallrye.mutiny.operators.multi.MultiOnRequestInvoke$MultiOnRequestInvokeOperator.request(MultiOnRequestInvoke.java:34)
at io.smallrye.reactive.messaging.amqp.AmqpCreditBasedSender.setCreditsAndRequest(AmqpCreditBasedSender.java:163)
at io.smallrye.reactive.messaging.amqp.AmqpCreditBasedSender.lambda$onNoMoreCredit$7(AmqpCreditBasedSender.java:218)
at io.vertx.mutiny.core.Context.lambda$runOnContext$1(Context.java:136)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252)
at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:50)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
This seems an internal wrong calculation that is forcefully closing the client connection.
To provide some more context after some investigation, the issue seems to happen when we send a huge amount of messages (e.g., 16k multiple times and in short period) which are sent independently and in different TX after completion callbacks (i.e., each message is sent from a different async callback) https://github.com/Hyperfoil/Horreum/blob/7092af13b6310ad6475a972745f4c3ede6f5dc4d/horreum-backend/src/main/java/io/hyperfoil/tools/horreum/svc/SchemaServiceImpl.java#L879-L882
Therefore my guess is that those events are pushed asynchronously when the DB transaction is committed and causing some sort of race condition in the messaging client library, when computing the (remaining?) credits.
Unfortunately I am not that expert on this, any help? or guidance?
Expected behavior
I am expecting the credits to be correctly computed
Actual behavior
Somehow the credits are lower than 0: Invalid request number, must be greater than 0 and that forces the client to stop the connection.
How to Reproduce?
I couldn't find any small reproducer at the moment
Output of uname -a or ver
Linux 8cef02ca36a0 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
Output of java -version
openjdk version "17.0.15" 2025-04-15 LTS OpenJDK Runtime Environment (Red_Hat-17.0.15.0.6-1) (build 17.0.15+6-LTS) OpenJDK 64-Bit Server VM (Red_Hat-17.0.15.0.6-1) (build 17.0.15+6-LTS, mixed mode, sharing)
Quarkus version or git rev
3.21.0
Build tool (ie. output of mvnw --version or gradlew --version)
Apache Maven 3.8.5 (Red Hat 3.8.5-6)
Additional information
Related to this Hyperfoil/Horreum#2425