ODistributedLockException on cluster start up #10589
Description
OrientDB Version: 3.2.45
Dockerfile extends from: eclipse-temurin:8-jdk
Background Info
It is a local dev cluster running on one machine with three master nodes in a docker environment. A database exists with some entries and only few kb of storage is used. No connection hits the database on startup.
After the exception was thrown it takes a small amount of time and everything works fine.
Steps to Reproduce
Spin up a local docker cluster with three orientdb container, create a database with some entries and watch the logs when the containers start.
It happens 95% of the time when starting the database, which was always shut down gracefully beforehand.
This is the exception
dev_odb2-1 | 2025-11-02 16:43:45:990 SEVER Exception in thread 'OrientDBEmbedded-3' [ScalingThreadPoolExecutor]
dev_odb2-1 | com.orientechnologies.orient.server.distributed.task.ODistributedLockException: Timed out after 20000 ms attempting to obtain lock for resource 'dev_project_main' on node 'dev-dev_odb2-1' DB name="dev_project_main"
dev_odb2-1 | at com.orientechnologies.orient.server.hazelcast.OHazelcastLockManager.acquireExclusiveLock(OHazelcastLockManager.java:39)
dev_odb2-1 | at com.orientechnologies.orient.server.hazelcast.OHazelcastClusterMetadataManager.executeInDistributedDatabaseLock(OHazelcastClusterMetadataManager.java:1400)
dev_odb2-1 | at com.orientechnologies.orient.server.distributed.impl.ODistributedPlugin.executeInDistributedDatabaseLock(ODistributedPlugin.java:264)
dev_odb2-1 | at com.orientechnologies.orient.server.distributed.impl.ODistributedPlugin.checkNodeInConfiguration(ODistributedPlugin.java:1357)
dev_odb2-1 | at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.initFirstOpen(ODistributedDatabaseImpl.java:604)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.newDistributedDatabase(OrientDBDistributed.java:510)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.lambda$registerNewDatabaseIfNeeded$1(OrientDBDistributed.java:533)
dev_odb2-1 | at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.registerNewDatabaseIfNeeded(OrientDBDistributed.java:533)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.newSessionInstance(OrientDBDistributed.java:166)
dev_odb2-1 | at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:447)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.openNoAuthorization(OrientDBDistributed.java:415)
dev_odb2-1 | at com.orientechnologies.orient.core.db.OrientDBEmbedded.lambda$executeNoAuthorization$8(OrientDBEmbedded.java:1137)
dev_odb2-1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
dev_odb2-1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
dev_odb2-1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
dev_odb2-1 | at java.lang.Thread.run(Thread.java:750)
This is the detailed log
dev_odb1-1 | 2025-11-02 16:43:07:328 INFO [dev-dev_odb1-1] Distributed servers status (*=current):
dev_odb1-1 |
dev_odb1-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb1-1 | |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
dev_odb1-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb1-1 | |dev-dev_odb3-1 |ONLINE|dev_project_main=ONLINE (MASTER) |0 |16:42:51 |15.15.0.22:2424|15.15.0.22:2480|488.10MB/3.83GB (12.43%)|
dev_odb1-1 | |dev-dev_odb2-1 |ONLINE|dev_project_main=NOT_AVAILABLE (MASTER)|2 |16:42:51 |15.15.0.21:2424|15.15.0.21:2480|498.33MB/3.83GB (12.69%)|
dev_odb1-1 | |dev-dev_odb1-1(*)|ONLINE|dev_project_main=ONLINE (MASTER) |4 |16:42:51 |15.15.0.20:2424|15.15.0.20:2480|251.58MB/3.83GB (6.41%) |
dev_odb1-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb2-1 | 2025-11-02 16:43:05:221 INFO Storage 'plocal:/orientdb/databases/dev_project_main' is opened under OrientDB distribution : 3.2.45 (build ${buildNumber}, branch UNKNOWN) [OEnterpriseLocalPaginatedStorage]
dev_odb2-1 | 2025-11-02 16:43:07:329 INFO [dev-dev_odb2-1] Distributed servers status (*=current):
dev_odb2-1 |
dev_odb2-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb2-1 | |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
dev_odb2-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb2-1 | |dev-dev_odb3-1 |ONLINE|dev_project_main=ONLINE (MASTER) |0 |16:42:51 |15.15.0.22:2424|15.15.0.22:2480|488.10MB/3.83GB (12.43%)|
dev_odb2-1 | |dev-dev_odb2-1(*)|ONLINE|dev_project_main=NOT_AVAILABLE (MASTER)|2 |16:42:51 |15.15.0.21:2424|15.15.0.21:2480|498.33MB/3.83GB (12.69%)|
dev_odb2-1 | |dev-dev_odb1-1 |ONLINE|dev_project_main=ONLINE (MASTER) |4 |16:42:51 |15.15.0.20:2424|15.15.0.20:2480|251.58MB/3.83GB (6.41%) |
dev_odb2-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb3-1 | 2025-11-02 16:43:04:878 INFO [dev-dev_odb3-1] Opening database 'dev_project_main'... [OrientDBDistributed]
dev_odb3-1 | 2025-11-02 16:43:07:329 INFO [dev-dev_odb3-1] Distributed servers status (*=current):
dev_odb3-1 |
dev_odb3-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb3-1 | |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
dev_odb3-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb3-1 | |dev-dev_odb3-1(*)|ONLINE|dev_project_main=ONLINE (MASTER) |0 |16:42:51 |15.15.0.22:2424|15.15.0.22:2480|488.10MB/3.83GB (12.43%)|
dev_odb3-1 | |dev-dev_odb2-1 |ONLINE|dev_project_main=NOT_AVAILABLE (MASTER)|2 |16:42:51 |15.15.0.21:2424|15.15.0.21:2480|498.33MB/3.83GB (12.69%)|
dev_odb3-1 | |dev-dev_odb1-1 |ONLINE|dev_project_main=ONLINE (MASTER) |4 |16:42:51 |15.15.0.20:2424|15.15.0.20:2480|251.58MB/3.83GB (6.41%) |
dev_odb3-1 | +-----------------+------+--------------------------------------------+-----+---------+---------------+---------------+------------------------+
dev_odb3-1 | [OHazelcastPlugin]
dev_odb2-1 | [OHazelcastPlugin]
dev_odb2-1 | 2025-11-02 16:43:45:990 SEVER Exception in thread 'OrientDBEmbedded-3' [ScalingThreadPoolExecutor]
dev_odb2-1 | com.orientechnologies.orient.server.distributed.task.ODistributedLockException: Timed out after 20000 ms attempting to obtain lock for resource 'dev_project_main' on node 'dev-dev_odb2-1' DB name="dev_project_main"
dev_odb2-1 | at com.orientechnologies.orient.server.hazelcast.OHazelcastLockManager.acquireExclusiveLock(OHazelcastLockManager.java:39)
dev_odb2-1 | at com.orientechnologies.orient.server.hazelcast.OHazelcastClusterMetadataManager.executeInDistributedDatabaseLock(OHazelcastClusterMetadataManager.java:1400)
dev_odb2-1 | at com.orientechnologies.orient.server.distributed.impl.ODistributedPlugin.executeInDistributedDatabaseLock(ODistributedPlugin.java:264)
dev_odb2-1 | at com.orientechnologies.orient.server.distributed.impl.ODistributedPlugin.checkNodeInConfiguration(ODistributedPlugin.java:1357)
dev_odb2-1 | at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.initFirstOpen(ODistributedDatabaseImpl.java:604)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.newDistributedDatabase(OrientDBDistributed.java:510)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.lambda$registerNewDatabaseIfNeeded$1(OrientDBDistributed.java:533)
dev_odb2-1 | at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.registerNewDatabaseIfNeeded(OrientDBDistributed.java:533)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.newSessionInstance(OrientDBDistributed.java:166)
dev_odb2-1 | at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:447)
dev_odb2-1 | at com.orientechnologies.orient.distributed.db.OrientDBDistributed.openNoAuthorization(OrientDBDistributed.java:415)
dev_odb2-1 | at com.orientechnologies.orient.core.db.OrientDBEmbedded.lambda$executeNoAuthorization$8(OrientDBEmbedded.java:1137)
dev_odb2-1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
dev_odb2-1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
dev_odb2-1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
dev_odb2-1 | at java.lang.Thread.run(Thread.java:750)
dev_odb2-1 |
dev_odb2-1 | 2025-11-02 16:43:45:997 WARNI [dev-dev_odb2-1]->[[dev-dev_odb3-1]] requesting delta database sync for 'dev_project_main' on local server... [OHazelcastPlugin]
dev_odb2-1 | 2025-11-02 16:43:46:006 INFO [dev-dev_odb2-1] Publishing ONLINE status for database dev-dev_odb2-1.dev_project_main... [ODistributedDatabaseImpl]
dev_odb1-1 | [OHazelcastPlugin]
dev_odb3-1 | 2025-11-02 16:43:14:443 INFO [dev-dev_odb3-1] Server 'dev-dev_odb2-1' was not found in the list of registered servers. Reloading configuration from cluster... [OClusterHealthChecker]
dev_odb3-1 | 2025-11-02 16:43:46:006 INFO [dev-dev_odb3-1]<-[dev-dev_odb2-1] Received new status dev-dev_odb2-1.dev_project_main=ONLINE [OHazelcastClusterMetadataManager]
dev_odb3-1 | 2025-11-02 16:43:46:007 INFO [dev-dev_odb3-1] Distributed servers status (*=current):
dev_odb3-1 |
dev_odb3-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb3-1 | |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
dev_odb3-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb3-1 | |dev-dev_odb3-1(*)|ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.22:2424|15.15.0.22:2480|239.89MB/3.83GB (6.11%)|
dev_odb3-1 | |dev-dev_odb2-1 |ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.21:2424|15.15.0.21:2480|141.65MB/3.83GB (3.61%)|
dev_odb3-1 | |dev-dev_odb1-1 |ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.20:2424|15.15.0.20:2480|264.34MB/3.83GB (6.73%)|
dev_odb3-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb1-1 | 2025-11-02 16:43:46:006 INFO [dev-dev_odb1-1]<-[dev-dev_odb2-1] Received new status dev-dev_odb2-1.dev_project_main=ONLINE [OHazelcastClusterMetadataManager]
dev_odb2-1 | 2025-11-02 16:43:46:006 INFO [dev-dev_odb2-1] Received new status dev-dev_odb2-1.dev_project_main=ONLINE [OHazelcastClusterMetadataManager]
dev_odb2-1 | 2025-11-02 16:43:46:007 INFO [dev-dev_odb2-1] Distributed servers status (*=current):
dev_odb2-1 |
dev_odb2-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb2-1 | |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
dev_odb2-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb2-1 | |dev-dev_odb3-1 |ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.22:2424|15.15.0.22:2480|239.89MB/3.83GB (6.11%)|
dev_odb2-1 | |dev-dev_odb2-1(*)|ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.21:2424|15.15.0.21:2480|141.65MB/3.83GB (3.61%)|
dev_odb2-1 | |dev-dev_odb1-1 |ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.20:2424|15.15.0.20:2480|264.34MB/3.83GB (6.73%)|
dev_odb2-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb1-1 | 2025-11-02 16:43:46:008 INFO [dev-dev_odb1-1] Distributed servers status (*=current):
dev_odb1-1 |
dev_odb1-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb1-1 | |Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
dev_odb1-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb1-1 | |dev-dev_odb3-1 |ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.22:2424|15.15.0.22:2480|239.89MB/3.83GB (6.11%)|
dev_odb1-1 | |dev-dev_odb2-1 |ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.21:2424|15.15.0.21:2480|141.65MB/3.83GB (3.61%)|
dev_odb1-1 | |dev-dev_odb1-1(*)|ONLINE|dev_project_main=ONLINE (MASTER)|4 |16:42:51 |15.15.0.20:2424|15.15.0.20:2480|264.34MB/3.83GB (6.73%)|
dev_odb1-1 | +-----------------+------+-------------------------------------+-----+---------+---------------+---------------+-----------------------+
dev_odb2-1 | [OHazelcastPlugin]
dev_odb2-1 | 2025-11-02 16:43:46:008 INFO {db=dev_project_main} [dev-dev_odb2-1] Reassigning ownership of clusters for database dev_project_main... [OHazelcastPlugin]
dev_odb2-1 | 2025-11-02 16:43:46:015 INFO {db=dev_project_main} [dev-dev_odb2-1] Reassignment of clusters for database 'dev_project_main' completed (classes=44) [OHazelcastPlugin]
dev_odb2-1 | 2025-11-02 16:43:46:016 INFO {db=dev_project_main} [dev-dev_odb2-1] Setting new distributed configuration for database: dev_project_main (version=422)
dev_odb2-1 | [ODistributedConfigurationManager]
dev_odb2-1 | 2025-11-02 16:43:46:047 INFO [dev-dev_odb2-1] Broadcasting new distributed configuration for database: dev_project_main (version=422)
dev_odb1-1 | [OHazelcastPlugin]
dev_odb1-1 | 2025-11-02 16:43:46:064 INFO {db=dev_project_main} [dev-dev_odb1-1] Setting new distributed configuration for database: dev_project_main (version=422)
dev_odb3-1 | [OHazelcastPlugin]
dev_odb3-1 | 2025-11-02 16:43:46:066 INFO {db=dev_project_main} [dev-dev_odb3-1] Setting new distributed configuration for database: dev_project_main (version=422)
dev_odb2-1 | [OHazelcastClusterMetadataManager]