A three-node database, continuously updating the same data across different nodes, and one node crashes, why? #10551
Description
orientdb version 3.2.29
2025-09-11 12:09:14:067 FINE {db=cmdb} Created record #1097:3 v.1 size=52 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:067 FINE {db=cmdb} Created record #1097:2 v.1 size=50 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:067 FINE {db=cmdb} Created record #1097:1 v.1 size=52 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:067 FINE {db=cmdb} 64 Committed transaction 3019 on database 'cmdb' (result=[ORecordOperation [record=#1097:3{ip_inet:1.68427888E8,ip:10.10.1.112,originRid:#586:0} v1, type=CREATE], ORecordOperation [record=#1097:2{ip_inet:1.67838014E8,ip:10.1.1.62,originRid:#681:0} v1, type=CREATE], ORecordOperation [record=#1097:1{ip_inet:1.68427888E8,ip:10.10.1.112,originRid:#593:0} v1, type=CREATE]]) [OLocalPaginatedStorage]
2025-09-11 12:09:14:067 FINE {db=cmdb} Read record #0:1 v.171593 size=64737 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:073 FINE {db=cmdb} Updated record #0:1 v.171594 size=64737 [OLocalPaginatedStorage]
2025-09-11 12:09:14:074 FINE {db=cmdb} 64 Committed transaction 3020 on database 'cmdb' (result=[ORecordOperation [record=#0:1{schemaVersion:4,classes:[156],views:[1],globalProperties:[49],blobClusters:[0]} v171594, type=UPDATE]]) [OLocalPaginatedStorage]
2025-09-11 12:09:14:077 INFO {db=cmdb} Replaced for view 'ipManage' clusters '[1105 => v_7_ipmanage]' with '1097 => v_3_ipmanage' [ViewManager]
2025-09-11 12:09:14:077 INFO {db=cmdb} Replaced for view 'ipManage' indexes '[]' with '[]' [ViewManager]
2025-09-11 12:09:14:077 INFO {db=cmdb} Finished refresh of view 'ipManage' [ViewManager]
2025-09-11 12:09:14:079 FINE {db=cmdb} Read record #1105:0 v.1 size=52 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:079 FINE {db=cmdb} Deleted record #1105:0 v.1 [OLocalPaginatedStorage]
2025-09-11 12:09:14:079 FINE {db=cmdb} 64 Committed transaction 3021 on database 'cmdb' (result=[ORecordOperation [record=ipManage#1105:0{ip_inet:1.6843009E8,ip:10.10.10.10,originRid:#617:0} v1, type=DELETE]]) [OLocalPaginatedStorage]
2025-09-11 12:09:14:079 FINE {db=cmdb} Read record #1105:1 v.1 size=52 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:079 FINE {db=cmdb} Deleted record #1105:1 v.1 [OLocalPaginatedStorage]
2025-09-11 12:09:14:080 FINE {db=cmdb} 64 Committed transaction 3022 on database 'cmdb' (result=[ORecordOperation [record=ipManage#1105:1{ip_inet:1.68427888E8,ip:10.10.1.112,originRid:#593:0} v1, type=DELETE]]) [OLocalPaginatedStorage]
2025-09-11 12:09:14:080 FINE {db=cmdb} Read record #1105:2 v.1 size=50 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:080 FINE {db=cmdb} Deleted record #1105:2 v.1 [OLocalPaginatedStorage]
2025-09-11 12:09:14:080 FINE {db=cmdb} 64 Committed transaction 3023 on database 'cmdb' (result=[ORecordOperation [record=ipManage#1105:2{ip_inet:1.67838014E8,ip:10.1.1.62,originRid:#681:0} v1, type=DELETE]]) [OLocalPaginatedStorage]
2025-09-11 12:09:14:080 FINE {db=cmdb} Read record #1105:3 v.1 size=52 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:14:080 FINE {db=cmdb} Deleted record #1105:3 v.1 [OLocalPaginatedStorage]
2025-09-11 12:09:14:080 FINE {db=cmdb} 64 Committed transaction 3024 on database 'cmdb' (result=[ORecordOperation [record=ipManage#1105:3{ip_inet:1.68427888E8,ip:10.10.1.112,originRid:#586:0} v1, type=DELETE]]) [OLocalPaginatedStorage]
2025-09-11 12:09:14:286 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:14:286 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:14:286 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:14:349 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Cannot publish member list to cluster. Is-master: false, joined: true , mastership claim in progress: false [MembershipManager]
2025-09-11 12:09:14:506 FINE {db=OSystem} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:14:506 FINE {db=OSystem} onClose [OLuceneIndexFactory]
2025-09-11 12:09:14:684 FINE {db=cmdb} Transaction with ID 3017 was already started 7 times, and will be reused. [OTransactionOptimisticDistributed]
2025-09-11 12:09:15:228 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:15:229 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:15:507 FINE {db=OSystem} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:15:507 FINE {db=OSystem} onClose [OLuceneIndexFactory]
2025-09-11 12:09:15:525 FINE {db=cmdb} Transaction with ID 3017 was already started 8 times, and will be reused. [OTransactionOptimisticDistributed]
2025-09-11 12:09:15:732 FINE {db=cmdb} Transaction with ID 3017 was already started 9 times, and will be reused. [OTransactionOptimisticDistributed]
2025-09-11 12:09:16:138 FINE Request id:174 type:120 [ONetworkProtocolBinary]
2025-09-11 12:09:16:140 FINE Request id:174 type:120 [ONetworkProtocolBinary]
2025-09-11 12:09:16:146 INFO [30] Current node is a MASTER for database 'cmdb' [OHazelcastPlugin]
2025-09-11 12:09:16:146 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:16:146 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:229 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:16:229 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:424 FINE {db=cmdb} Transaction with ID 3017 was already started 10 times, and will be reused. [OTransactionOptimisticDistributed]
2025-09-11 12:09:16:425 WARNI {db=cmdb} Error beginning timed out transaction: 1.152039 [ODatabaseDocumentDistributed]
com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #561:8 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v76284 your=v76285)
DB name="cmdb"
Error Code="3"
at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.firstPhaseDataChecks(ODatabaseDocumentDistributed.java:989)
at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.internalBegin2pc(ODatabaseDocumentDistributed.java:867)
at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.commit2pc(ODatabaseDocumentDistributed.java:735)
at com.orientechnologies.orient.server.distributed.impl.task.OTransactionPhase2Task.execute(OTransactionPhase2Task.java:127)
at com.orientechnologies.orient.server.distributed.impl.ODistributedPlugin.localInternalExecute(ODistributedPlugin.java:983)
at com.orientechnologies.orient.server.distributed.impl.ODistributedPlugin.lambda$executeOnLocalNode$2(ODistributedPlugin.java:973)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:91)
at com.orientechnologies.orient.server.distributed.impl.ODistributedPlugin.executeOnLocalNode(ODistributedPlugin.java:971)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.execute(ODistributedDatabaseImpl.java:369)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.lambda$null$5(ODistributedDatabaseImpl.java:321)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2025-09-11 12:09:16:426 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:426 WARNI Reached limit of retry for commit tx:1.152039 forcing database re-install [ODatabaseDocumentDistributed]
2025-09-11 12:09:16:426 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:16:427 FINE {db=cmdb} Updated record #561:8 v.76285 size=415 [OLocalPaginatedStorage]
2025-09-11 12:09:16:427 FINE {db=cmdb} 79 Committed transaction 3016 on database 'cmdb' (result=[ORecordOperation [record=Yewuxitong#561:8{in_BasicModelInstanceRelationship:[#200:277],out_CustomerDepartmentRelationship:[#427:0],out_CustomerbusinessDepartmentRelationship:[#982:0],in_CustomerApplicationRelationship:[#337:1],out_CustomerpeizhixiangguanliyuanRelationship:[],out_CustomerPicRelationship:[#386:4],createTime:2025-09-11 12:09:11,name:通达信快订,appLevel:IV级,updateTime:2025-09-11 12:09:11,state:0,uuid:24d4c3a6453444c589ec48e8abb2b731,licenseCount:20} v76285, type=UPDATE]]) [OLocalPaginatedStorage]
2025-09-11 12:09:16:427 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:427 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:16:427 FINE {db=cmdb} Read record #561:8 v.76285 size=415 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:16:428 INFO {db=cmdb} Persistent version not match, record:#561:8 expected:76286 actual:76285 waiting for re-enqueue request [ODatabaseDocumentDistributed]
2025-09-11 12:09:16:428 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:428 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:16:428 INFO {db=cmdb} Received second phase but not yet first phase for commit tx:1.152041, re-enqueue second phase [OTransactionPhase2Task]
2025-09-11 12:09:16:428 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:429 WARNI [30]->[[node64]] requesting delta database sync for 'cmdb' on local server... [OHazelcastPlugin]
2025-09-11 12:09:16:429 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:16:430 FINE {db=cmdb} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:437 FINE Request id:170 type:121 [ONetworkProtocolBinary]
2025-09-11 12:09:16:437 INFO [30] Requesting full sync for database 'cmdb'... [OHazelcastPlugin]
2025-09-11 12:09:16:437 INFO [30]->[[node64]] Requesting deploy of database 'cmdb' on local server... [OHazelcastPlugin]
2025-09-11 12:09:16:441 INFO [30]<-[node64] Received updated status 30.cmdb=SYNCHRONIZING [OHazelcastClusterMetadataManager]
2025-09-11 12:09:16:442 INFO [30] Distributed servers status (*=current):
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
|Name |Status|Databases |Conns|StartedOn |Binary |HTTP |UsedMemory |
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
|node64|ONLINE|cmdb=ONLINE (MASTER) |464 |2025-09-08 09:48:53.721|10.11.2.64:2424|10.11.2.64:2480|1.66GB/1.97GB (84.40%) |
|30(*) |ONLINE|cmdb=SYNCHRONIZING (MASTER)|61 |10:38:12 |10.10.1.30:2424|10.10.1.30:2480|295.96MB/1.78GB (16.26%)|
|node63|ONLINE|cmdb=ONLINE (MASTER) |67 |10:40:16 |10.11.2.63:2424|10.11.2.63:2480|126.23MB/2.00GB (6.16%) |
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
[OHazelcastPlugin]
2025-09-11 12:09:16:451 INFO [30]<-[node64] Received updated status node64.cmdb=BACKUP [OHazelcastClusterMetadataManager]
2025-09-11 12:09:16:452 INFO [30] Distributed servers status (*=current):
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
|Name |Status|Databases |Conns|StartedOn |Binary |HTTP |UsedMemory |
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
|node64|ONLINE|cmdb=BACKUP (MASTER) |464 |2025-09-08 09:48:53.721|10.11.2.64:2424|10.11.2.64:2480|1.66GB/1.97GB (84.40%) |
|30(*) |ONLINE|cmdb=SYNCHRONIZING (MASTER)|61 |10:38:12 |10.10.1.30:2424|10.10.1.30:2480|295.96MB/1.78GB (16.26%)|
|node63|ONLINE|cmdb=ONLINE (MASTER) |67 |10:40:16 |10.11.2.63:2424|10.11.2.63:2480|126.23MB/2.00GB (6.16%) |
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
[OHazelcastPlugin]
2025-09-11 12:09:16:506 FINE {db=OSystem} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:16:506 FINE {db=OSystem} onClose [OLuceneIndexFactory]
2025-09-11 12:09:16:855 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Received heartbeat from Member [10.11.2.64]:2434 - 7ec56651-ec71-427a-89b2-8bf7bbee2c03 (now: 2025-09-11 12:09:16.853, timestamp: 2025-09-11 12:09:16.853) [ClusterHeartbeatManager]
2025-09-11 12:09:16:855 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Setting cluster time diff to -2ms. [ClusterService]
2025-09-11 12:09:17:229 FINE {db=cmdb} onOpen [OLuceneIndexFactory]
2025-09-11 12:09:17:229 FINE {db=cmdb} onClose [OLuceneIndexFactory]Exception in thread "OrientDB Timer[embedded:/usr/local/db/orientdb-3.2.29/databases]" com.orientechnologies.common.concur.OOfflineNodeException: Node is offline
at com.orientechnologies.orient.server.distributed.impl.lock.OLockManagerImpl.lock(OLockManagerImpl.java:37)
at com.orientechnologies.orient.server.distributed.impl.lock.OLockManagerImpl.lock(OLockManagerImpl.java:73)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.processRequest(ODistributedDatabaseImpl.java:335)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl$1.run(ODistributedDatabaseImpl.java:273)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
2025-09-11 12:09:17:827 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Received heartbeat from Member [10.11.2.63]:2434 - a21eb3f0-e0a0-4dac-943d-f24e7b1f78e0 (now: 2025-09-11 12:09:17.825, timestamp: 2025-09-11 12:09:17.827) [ClusterHeartbeatManager]
2025-09-11 12:09:21:672 FINE Request id:170 type:121 [ONetworkProtocolBinary]
2025-09-11 12:09:21:766 INFO [30]<-[node64] Installing database 'cmdb' to: /usr/local/db/orientdb-3.2.29/databases/cmdb... [OHazelcastPlugin]
2025-09-11 12:09:21:766 INFO [30] - writing chunk #1 offset=0 size=8.00MB [OSyncReceiver]
2025-09-11 12:09:21:854 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Received heartbeat from Member [10.11.2.64]:2434 - 7ec56651-ec71-427a-89b2-8bf7bbee2c03 (now: 2025-09-11 12:09:21.852, timestamp: 2025-09-11 12:09:21.853) [ClusterHeartbeatManager]
2025-09-11 12:09:21:854 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Setting cluster time diff to -1ms. [ClusterService]
2025-09-11 12:09:22:671 FINE Request id:171 type:120 [ONetworkProtocolBinary]
2025-09-11 12:09:22:673 INFO [30]<-[node64] Received updated status node64.cmdb=ONLINE [OHazelcastClusterMetadataManager]
2025-09-11 12:09:22:675 INFO [30] Distributed servers status (*=current):
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
|Name |Status|Databases |Conns|StartedOn |Binary |HTTP |UsedMemory |
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
|node64|ONLINE|cmdb=ONLINE (MASTER) |465 |2025-09-08 09:48:53.721|10.11.2.64:2424|10.11.2.64:2480|1.74GB/1.97GB (88.45%) |
|30(*) |ONLINE|cmdb=SYNCHRONIZING (MASTER)|61 |10:38:12 |10.10.1.30:2424|10.10.1.30:2480|295.96MB/1.78GB (16.26%)|
|node63|ONLINE|cmdb=ONLINE (MASTER) |67 |10:40:16 |10.11.2.63:2424|10.11.2.63:2480|126.23MB/2.00GB (6.16%) |
+------+------+---------------------------+-----+-----------------------+---------------+---------------+------------------------+
[OHazelcastPlugin]
2025-09-11 12:09:22:828 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Received heartbeat from Member [10.11.2.63]:2434 - a21eb3f0-e0a0-4dac-943d-f24e7b1f78e0 (now: 2025-09-11 12:09:22.827, timestamp: 2025-09-11 12:09:22.826) [ClusterHeartbeatManager]
2025-09-11 12:09:24:118 INFO [10.10.1.30]:2434 [orientdb] [3.12.13] processors=4, physical.memory.total=15.7G, physical.memory.free=3.8G, swap.space.total=1024.0M, swap.space.free=1012.2M, heap.memory.used=360.4M, heap.memory.free=1.1G, heap.memory.total=1.5G, heap.memory.max=1.8G, heap.memory.used/total=23.98%, heap.memory.used/max=19.79%, minor.gc.count=131, minor.gc.time=1754ms, major.gc.count=3, major.gc.time=287ms, load.process=0.00%, load.system=0.00%, load.systemAverage=0.08, thread.count=58, thread.peakCount=76, cluster.timeDiff=-1, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=39143, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=2, client.connection.count=0, connection.count=2 [HealthMonitor]
2025-09-11 12:09:26:137 FINE Request id:174 type:120 [ONetworkProtocolBinary]
2025-09-11 12:09:26:854 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Received heartbeat from Member [10.11.2.64]:2434 - 7ec56651-ec71-427a-89b2-8bf7bbee2c03 (now: 2025-09-11 12:09:26.853, timestamp: 2025-09-11 12:09:26.853) [ClusterHeartbeatManager]
2025-09-11 12:09:26:854 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Setting cluster time diff to -1ms. [ClusterService]
2025-09-11 12:09:27:400 FINE Request id:170 type:121 [ONetworkProtocolBinary]
2025-09-11 12:09:27:435 INFO [30] - writing chunk #2 offset=0 size=3.55MB [OSyncReceiver]
2025-09-11 12:09:27:637 INFO Page size for WAL located in /usr/local/db/orientdb-3.2.29/databases/cmdb is set to 4096 bytes. [CASDiskWriteAheadLog]
2025-09-11 12:09:27:828 FINE [10.10.1.30]:2434 [orientdb] [3.12.13] Received heartbeat from Member [10.11.2.63]:2434 - a21eb3f0-e0a0-4dac-943d-f24e7b1f78e0 (now: 2025-09-11 12:09:27.826, timestamp: 2025-09-11 12:09:27.826) [ClusterHeartbeatManager]
2025-09-11 12:09:28:526 INFO DWL:cmdb: block size = 4096 bytes, maximum segment size = 575 MB [DoubleWriteLogGL]
2025-09-11 12:09:28:526 SEVER File with startup metadata is broken and can not be used, creation of new one [StorageStartupMetadata]
2025-09-11 12:09:28:608 WARNI Storage 'cmdb' was not closed properly. Will try to recover from write ahead log [OLocalPaginatedStorage]
2025-09-11 12:09:28:608 INFO Looking for last checkpoint... [OLocalPaginatedStorage]
2025-09-11 12:09:28:609 INFO Data restore procedure is started. [OLocalPaginatedStorage]
2025-09-11 12:09:28:625 INFO [30] Database copied correctly, size=11.55MB [OSyncReceiver]
2025-09-11 12:09:28:627 WARNI Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.common.EmptyWALRecord{lsn =OLogSequenceNumber{segment=363, position=22}} will be skipped during data restore [OLocalPaginatedStorage]
2025-09-11 12:09:28:627 INFO 1 operations were processed, current LSN is OLogSequenceNumber{segment=363, position=22} last LSN is OLogSequenceNumber{segment=364, position=22} [OLocalPaginatedStorage]
2025-09-11 12:09:28:627 WARNI Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.common.EmptyWALRecord{lsn =OLogSequenceNumber{segment=364, position=22}} will be skipped during data restore [OLocalPaginatedStorage]
2025-09-11 12:09:28:752 INFO Storage data recover was completed [OLocalPaginatedStorage]
2025-09-11 12:09:28:933 FINE VPM open fileId:6069385092: fileName = OFunction.name.vpm [OVersionPositionMapV0]
2025-09-11 12:09:28:934 FINE VPM open fileId:5699398539: fileName = ORole.name.vpm [OVersionPositionMapV0]
2025-09-11 12:09:28:934 FINE VPM open fileId:4332118164: fileName = OUser.name.vpm [OVersionPositionMapV0]
2025-09-11 12:09:28:935 FINE VPM open fileId:5916301504: fileName = dictionary.vpm [OVersionPositionMapV0]
2025-09-11 12:09:28:935 FINE VPM open fileId:6186450092: fileName = OSecurityPolicy.name.vpm [OVersionPositionMapV0]
2025-09-11 12:09:28:941 INFO Storage 'plocal:/usr/local/db/orientdb-3.2.29/databases/cmdb' is opened under OrientDB distribution : 3.2.29 (build ${buildNumber}, branch UNKNOWN) [OLocalPaginatedStorage]
2025-09-11 12:09:28:942 INFO [30] Publishing ONLINE status for database 30.cmdb... [ODistributedDatabaseImpl]
2025-09-11 12:09:28:943 INFO [30] Received updated status 30.cmdb=ONLINE [OHazelcastClusterMetadataManager]
2025-09-11 12:09:28:944 INFO [30] Distributed servers status (*=current):
+------+------+--------------------+-----+-----------------------+---------------+---------------+------------------------+
|Name |Status|Databases |Conns|StartedOn |Binary |HTTP |UsedMemory |
+------+------+--------------------+-----+-----------------------+---------------+---------------+------------------------+
|node64|ONLINE|cmdb=ONLINE (MASTER)|465 |2025-09-08 09:48:53.721|10.11.2.64:2424|10.11.2.64:2480|1.83GB/1.97GB (92.74%) |
|30(*) |ONLINE|cmdb=ONLINE (MASTER)|61 |10:38:12 |10.10.1.30:2424|10.10.1.30:2480|295.96MB/1.78GB (16.26%)|
|node63|ONLINE|cmdb=ONLINE (MASTER)|67 |10:40:16 |10.11.2.63:2424|10.11.2.63:2480|162.75MB/2.00GB (7.95%) |
+------+------+--------------------+-----+-----------------------+---------------+---------------+------------------------+
[OHazelcastPlugin]
2025-09-11 12:09:28:945 WARNI I/O Error on distributed channel clientId=171 reqType=120 [ONetworkProtocolBinary]
java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:397)
at java.util.Timer.schedule(Timer.java:248)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.schedule(OrientDBEmbedded.java:1118)
at com.orientechnologies.orient.server.distributed.impl.task.OTransactionPhase1Task.received(OTransactionPhase1Task.java:453)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.processRequest(ODistributedDatabaseImpl.java:307)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeDistributedRequest(ONetworkProtocolBinary.java:673)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.distributedRequest(ONetworkProtocolBinary.java:497)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:237)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:68)
2025-09-11 12:09:28:949 FINE {db=cmdb} Read record #0:1 v.171591 size=64737 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:972 FINE {db=cmdb} Read record #0:2 v.109 size=24312 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:984 FINE {db=cmdb} Read record #4:0 v.12 size=226 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:985 FINE {db=cmdb} Read record #1085:0 v.2 size=57 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:986 FINE {db=cmdb} Read record #4:1 v.15 size=584 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:987 FINE {db=cmdb} Read record #1085:1 v.2 size=61 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:989 FINE {db=cmdb} Read record #1085:2 v.2 size=62 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:989 FINE {db=cmdb} Read record #4:2 v.18 size=756 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:990 FINE {db=cmdb} Read record #1085:3 v.2 size=58 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:994 FINE {db=cmdb} Read record #6:1 v.1 size=377 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:28:997 FINE {db=cmdb} Read record #0:1 v.171591 size=64737 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:027 FINE {db=cmdb} Read record #0:2 v.109 size=24312 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:043 FINE {db=cmdb} Read record #4:0 v.12 size=226 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:043 FINE {db=cmdb} Read record #1085:0 v.2 size=57 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:044 FINE {db=cmdb} Read record #4:1 v.15 size=584 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:044 FINE {db=cmdb} Read record #1085:1 v.2 size=61 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:046 FINE {db=cmdb} Read record #1085:2 v.2 size=62 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:046 FINE {db=cmdb} Read record #4:2 v.18 size=756 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:047 FINE {db=cmdb} Read record #1085:3 v.2 size=58 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:049 FINE {db=cmdb} Read record #6:1 v.1 size=377 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:051 FINE {db=cmdb} Read record #0:1 v.171591 size=64737 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:051 SEVER Exception in thread 'OrientDBEmbedded-2' [ScalingThreadPoolExecutor]
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database 'cmdb'
at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:465)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.lambda$executeNoAuthorization$8(OrientDBEmbedded.java:1150)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database url=plocal:/usr/local/db/orientdb-3.2.29/databases/cmdb
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:216)
at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.lambda$init$0(ODatabaseDocumentDistributed.java:323)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:91)
at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.init(ODatabaseDocumentDistributed.java:321)
at com.orientechnologies.orient.distributed.db.OrientDBDistributed.newSessionInstance(OrientDBDistributed.java:129)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:459)
... 5 more
Caused by: java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:397)
at java.util.Timer.schedule(Timer.java:248)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.schedule(OrientDBEmbedded.java:1118)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.schedule(ViewManager.java:156)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.start(ViewManager.java:139)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.load(ViewManager.java:129)
at com.orientechnologies.orient.server.distributed.impl.metadata.OSharedContextDistributed.lambda$load$1(OSharedContextDistributed.java:95)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:91)
at com.orientechnologies.orient.server.distributed.impl.metadata.OSharedContextDistributed.load(OSharedContextDistributed.java:79)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.loadMetadata(ODatabaseDocumentEmbedded.java:347)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:203)
... 10 more
2025-09-11 12:09:29:072 FINE {db=cmdb} Read record #0:2 v.109 size=24312 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:086 FINE {db=cmdb} Read record #4:0 v.12 size=226 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:086 FINE {db=cmdb} Read record #1085:0 v.2 size=57 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:087 FINE {db=cmdb} Read record #4:1 v.15 size=584 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:088 FINE {db=cmdb} Read record #1085:1 v.2 size=61 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:089 FINE {db=cmdb} Read record #1085:2 v.2 size=62 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:089 FINE {db=cmdb} Read record #4:2 v.18 size=756 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:090 FINE {db=cmdb} Read record #1085:3 v.2 size=58 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:092 FINE {db=cmdb} Read record #6:1 v.1 size=377 bytes [OLocalPaginatedStorage]
2025-09-11 12:09:29:093 SEVER Exception in thread 'OrientDBEmbedded-4' [ScalingThreadPoolExecutor]
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database 'cmdb'
at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:465)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.lambda$executeNoAuthorization$8(OrientDBEmbedded.java:1150)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database url=plocal:/usr/local/db/orientdb-3.2.29/databases/cmdb
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:216)
at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.lambda$init$0(ODatabaseDocumentDistributed.java:323)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:91)
at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.init(ODatabaseDocumentDistributed.java:321)
at com.orientechnologies.orient.distributed.db.OrientDBDistributed.newSessionInstance(OrientDBDistributed.java:129)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:459)
... 5 more
Caused by: java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:397)
at java.util.Timer.schedule(Timer.java:248)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.schedule(OrientDBEmbedded.java:1118)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.schedule(ViewManager.java:156)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.start(ViewManager.java:139)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.load(ViewManager.java:129)
at com.orientechnologies.orient.server.distributed.impl.metadata.OSharedContextDistributed.lambda$load$1(OSharedContextDistributed.java:95)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:91)
at com.orientechnologies.orient.server.distributed.impl.metadata.OSharedContextDistributed.load(OSharedContextDistributed.java:79)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.loadMetadata(ODatabaseDocumentEmbedded.java:347)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:203)
... 10 more
[](url)