MongoClient should reduce the number of created threads

[franz1981]

Description

While profiling Quarkus super heroes we found that mongdb create a huge amount of Threads to handle async I/O (and other operations) - see the attached profiling data

FIGHTS_asprof_steady.zip

Implementation ideas

I see that https://mongodb.github.io/mongo-java-driver/4.1/apidocs/mongodb-driver-core/com/mongodb/MongoClientSettings.Builder.html#streamFactoryFactory(com.mongodb.connection.StreamFactoryFactory) allows to configure https://mongodb.github.io/mongo-java-driver/4.1/apidocs/mongodb-driver-core/com/mongodb/connection/netty/NettyStreamFactoryFactory.html which could even enable us to re-use the existing Netty/Vertx event loop for mongodb.

In alternative having the chance to size & configure the running thread pool for the mongoDB client would be of great help - unless I've misread how it should be done (@edeandrea as well).

[quarkus-bot]

/cc @loicmathieu (mongodb)

[franz1981]

@cescoffier as well

[franz1981]

quarkus/extensions/mongodb-client/runtime/src/main/java/io/quarkus/mongodb/runtime/MongoClients.java

Line 249 in 7b846d6

private MongoClientSettings createMongoConfiguration(String name, MongoClientConfig config, boolean isReactive) {

seems where the configuration happen - but the flamegraph provided shows other threads which seems related to how mutiny interact with the reactor core part (maybe?), see

@jponge

[jponge]

@franz1981 Mutiny 2.8.0 has a fix that was indirectly spotted with "reactive" Mongo: smallrye/smallrye-mutiny#1758

[geoand]

So if that fix makes any difference, we should be able to see it by checking main

[franz1981]

@geoand @jponge but that fix is not tuning the mongo client to use Netty under the hood so, maybe will just fix the creation of huge number of threads, but still will need I/O threads to use normal Java sockets over blocking API, no?

[geoand]

I assume, I have not checke

[jponge]

The fix that I mentioned applies to any call stack with a Multi involved, and where reactive Mongo is on the other hand. From the flamegraph it seems that it's not the case (one-shot async operation)

[franz1981]

@jponge any idea what could lead to creating so many threads?

[jponge]

From the graph as soon as you enter Mongo-land to perform a subscription it eats most of the CPU time. I'm not sure Mutiny is at fault here TBH, I only see a Uni::emitOn, but that should be a hop to a Quarkus-managed thread pool. Perhaps it's the RestEasy layer that should be looked at.

[franz1981]

Looking at the names of the worker threads it doesn't seem something we manage in quarkus (it is not an enhanced executor service) - so there is an high chance that switching to Netty and reusing our event loops will automagically solve it since there is not anymore any needs to jump off to other executors...

[franz1981]

I've (likely?) found why:

searching for sun/nio/ch/AsynchronousSocketChannelImpl.read it highlight the "elastic" pool which will handle async NIO calls (sigh!!!), below, in violet

which, despite seems irrelevant it harm badly RSS and performance since it is backed, as the issue explained, by a large amount of platform threads.
The write side instead, re sun/nio/ch/AsynchronousSocketChannelImpl.write happens mostly in one thread (i.e. [AsyncGetter-3-thread-1 tid=142]), but in smaller parts in the previously mentioned thread pools as well, see below

This kind of relate to the commnt at #46206 (comment) where switching the mongo-client (reactive) from NIO 2 (async I/O) to Netty would save tons of resources, likely reusing the existing Vertx event loops.

@cescoffier wdyt about it?

[franz1981]

Thinking about it would be great to not just let mongo client to run on Netty event loops but to expose the "right" event loop to be used local to the event loop which need to use it i.e. basically having a mongo client per each Vertx event loop and let the caller use the right one

Is it something which look feasible @jponge ? Who is responsible of our impl here?