[CRASH] All servers crash after a few weeks running #383
Description
Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:6379 [cluster](+0x137bb9) [0x560b9307ebb9]
keydb-server 0.0.0.0:6379 [cluster](beforeSleep(aeEventLoop*)+0x73) [0x560b92fa1053]
keydb-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x10b) [0x560b92f9d35b]
keydb-server 0.0.0.0:6379 [cluster](aeMain+0x43) [0x560b92f9d863]
keydb-server 0.0.0.0:6379 [cluster](workerThreadMain(void*)+0x6a) [0x560b92fa4d6a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f87a48956db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f87a45be71f]
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:6379 [cluster](+0x137bb9) [0x560b9307ebb9]
keydb-server 0.0.0.0:6379 [cluster](handleClientsWithPendingWrites(int, int)+0x3b5) [0x560b92fc61c5]
keydb-server 0.0.0.0:6379 [cluster](+0x59a95) [0x560b92fa0a95]
keydb-server 0.0.0.0:6379 [cluster](aeProcessCmd(aeEventLoop*, int, void*, int)+0xfa) [0x560b92f9ceaa]
keydb-server 0.0.0.0:6379 [cluster](ProcessEventCore+0x124) [0x560b92f9d104]
keydb-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x225) [0x560b92f9d475]
keydb-server 0.0.0.0:6379 [cluster](aeMain+0x43) [0x560b92f9d863]
keydb-server 0.0.0.0:6379 [cluster](workerThreadMain(void*)+0x6a) [0x560b92fa4d6a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f87a48956db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f87a45be71f]
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:6379 [cluster](+0x137bb9) [0x560b9307ebb9]
keydb-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x519) [0x560b92f9d769]
keydb-server 0.0.0.0:6379 [cluster](aeMain+0x43) [0x560b92f9d863]
keydb-server 0.0.0.0:6379 [cluster](workerThreadMain(void*)+0x6a) [0x560b92fa4d6a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f87a48956db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f87a45be71f]
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:6379 [cluster](+0x137bb9) [0x560b9307ebb9]
keydb-server 0.0.0.0:6379 [cluster](ProcessPendingAsyncWrites()+0x84) [0x560b92fc2994]
keydb-server 0.0.0.0:6379 [cluster](flushReplBacklogToClients()+0x425) [0x560b92fdccd5]
keydb-server 0.0.0.0:6379 [cluster](beforeSleep(aeEventLoop*)+0x30d) [0x560b92fa12ed]
keydb-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x10b) [0x560b92f9d35b]
keydb-server 0.0.0.0:6379 [cluster](aeMain+0x43) [0x560b92f9d863]
keydb-server 0.0.0.0:6379 [cluster](workerThreadMain(void*)+0x6a) [0x560b92fa4d6a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f87a48956db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f87a45be71f]
!!! ERROR: Deadlock detected !!!
14: (0x7f80fb29b48c) client
13: (0x560b9339d8a0) AE (global)
printing backtrace for thread 16
printing backtrace for thread 13
printing backtrace for thread 15
printing backtrace for thread 14
!!! KeyDB Will Now Crash !!!
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
8:14:M 16 Dec 2021 09:21:10.387 # ------------------------------------------------
8:14:M 16 Dec 2021 09:21:10.387 # !!! Software Failure. Press left mouse button to continue
8:14:M 16 Dec 2021 09:21:10.387 # Guru Meditation: Deadlock detected #fastlock.cpp:268
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:6379 [cluster](fastlock_sleep+0x3fa) [0x560b9307824a]
keydb-server 0.0.0.0:6379 [cluster](+0x137bb9) [0x560b9307ebb9]
keydb-server 0.0.0.0:6379 [cluster](ProcessPendingAsyncWrites()+0x84) [0x560b92fc2994]
keydb-server 0.0.0.0:6379 [cluster](flushReplBacklogToClients()+0x425) [0x560b92fdccd5]
keydb-server 0.0.0.0:6379 [cluster](beforeSleep(aeEventLoop*)+0x30d) [0x560b92fa12ed]
keydb-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x10b) [0x560b92f9d35b]
keydb-server 0.0.0.0:6379 [cluster](aeMain+0x43) [0x560b92f9d863]
keydb-server 0.0.0.0:6379 [cluster](workerThreadMain(void*)+0x6a) [0x560b92fa4d6a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f87a48956db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f87a45be71f]
------ INFO OUTPUT ------
Aditional information
- OS distribution and version: docker eqalpha/keydb:x86_64_v6.2.0 running on Ubuntu 18.04
- Steps to reproduce (if any)
After 13-14 weeks of cluster running, all the master servers fail with this error, after the master is changed to another slave server, the server fails after a few hours. Once all the servers are restarted it runs smoothly for other 13-14 weeks.