Pub/Sub : Large number of threads created leading to OutOfMemory

[talonx]

This is described in detail in netty/netty#7520 . Since I narrowed it down to being a PubSub problem, I'm logging it here. Please let me know if I should copy-paste the contents of the other issue here.

Turning off PubSub (and replacing it with an in-memory messaging system) in the same application leads to perfectly normal behaviour without any memory issues. I've also monitored this using jvisualvm, and one observation is that each time com.google.cloud.pubsub.v1.MessageDispatcher$2.run is called, lot of new threads get created and don't die. The heap dump also shows this, with around 37k threads.

PubSub version
0.26.0-beta

JVM version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

OS version
Linux 4.10.0-40-generic #44~16.04.1-Ubuntu SMP Thu Nov 9 15:37:44 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

[talonx]

After some investigation, this looks an error where our code was not closing PubSub resources properly, leading to an explosion of threads. Handling it resolved the problem. Thanks for all those who looked into this.

[nurgasemetey]

After some investigation, this looks an error where our code was not closing PubSub resources properly, leading to an explosion of threads. Handling it resolved the problem. Thanks for all those who looked into this.

@talonx could you share details?

[talonx]

@nurgasemetey It's been a while, so I don't recall the exact details, but we were not closing resources like SubscriptionAdminClient and TopicAdminClient. The PubSub Java API has also changed since then, so it's difficult to pinpoint the exact classes.

[talonx]

@JustinBeckwith What action is required from me here? The issue was closed in 2017.

[JustinBeckwith]

None required - apologies for the noise!