GremlinServerPlugin/ArcadeDB Server fails to start when referencing the same database folder with different file paths #1345
Description
ArcadeDB Version:
23.9.1, 23.10.1, 23.11.1-SNAPSHOT
OS and JDK Version:
Windows 10 10.0 - OpenJDK 64-Bit Server VM 17.0.8 (Temurin-17.0.8+7)
Expected behavior
Both ArcadeDB Server and Gremlin server start successfully
Actual behavior
Depending on the server configuration, either the ArcadeDB Server or the Gremlin server fails to start.
Steps to reproduce
All property files only use forward slashes in file paths, example (gremlin-server.properties):
gremlin.graph=com.arcadedb.gremlin.ArcadeGraph
gremlin.arcadedb.directory=./arcadedb/databases/graph
Scenario 1: No database was created yet and no default databases were defined in the server configuration (arcadedb.server.defaultDatabases). When starting up the embedded ArcadeDB Server, the databases are created as configured in the Gremlin config files, but then the following error occurs:
Exception in thread "main" com.arcadedb.exception.DatabaseOperationException: Error on creating new database instance
at com.arcadedb.database.EmbeddedDatabase.openInternal(EmbeddedDatabase.java:1798)
at com.arcadedb.database.EmbeddedDatabase.open(EmbeddedDatabase.java:185)
at com.arcadedb.database.DatabaseFactory.open(DatabaseFactory.java:76)
at com.arcadedb.server.ArcadeDBServer.getDatabase(ArcadeDBServer.java:457)
at com.arcadedb.server.ArcadeDBServer.getDatabase(ArcadeDBServer.java:314)
at com.arcadedb.server.ArcadeDBServer.loadDatabases(ArcadeDBServer.java:485)
at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:161)
[...]
Caused by: com.arcadedb.utility.LockException: Database 'graph' is locked by another process (path=C:\[...]\.\arcadedb\databases\graph)
at com.arcadedb.database.EmbeddedDatabase.lockDatabase(EmbeddedDatabase.java:1628)
at com.arcadedb.database.EmbeddedDatabase.checkForRecovery(EmbeddedDatabase.java:1730)
at com.arcadedb.database.EmbeddedDatabase.openInternal(EmbeddedDatabase.java:1776)
... 10 more
Caused by: java.nio.channels.OverlappingFileLockException
at java.base/sun.nio.ch.FileLockTable.checkList(FileLockTable.java:229)
at java.base/sun.nio.ch.FileLockTable.add(FileLockTable.java:123)
at java.base/sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1321)
at java.base/java.nio.channels.FileChannel.tryLock(FileChannel.java:1191)
at com.arcadedb.database.EmbeddedDatabase.lockDatabase(EmbeddedDatabase.java:1607)
... 12 more
Scenario 2: After restarting the ArcadeDB server (now with existing databases), the Gremlin server fails to start due to the same reason:
Graph [graph] configured at [./arcadedb/config/gremlin-server.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class com.arcadedb.gremlin.ArcadeGraph]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class com.arcadedb.gremlin.ArcadeGraph]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:84)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:72)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:106)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.addGraph(DefaultGraphManager.java:63)
at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:721)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:58)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:127)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:90)
at com.arcadedb.server.gremlin.GremlinServerPlugin.startService(GremlinServerPlugin.java:85)
at com.arcadedb.server.ArcadeDBServer.registerPlugins(ArcadeDBServer.java:234)
at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:158)
[...]
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80)
... 20 more
Caused by: com.arcadedb.exception.DatabaseOperationException: Error on creating new database instance
at com.arcadedb.database.EmbeddedDatabase.openInternal(EmbeddedDatabase.java:1798)
at com.arcadedb.database.EmbeddedDatabase.open(EmbeddedDatabase.java:185)
at com.arcadedb.database.DatabaseFactory.open(DatabaseFactory.java:76)
at com.arcadedb.database.DatabaseFactory.open(DatabaseFactory.java:68)
at com.arcadedb.gremlin.ArcadeGraph.<init>(ArcadeGraph.java:108)
at com.arcadedb.gremlin.ArcadeGraph.open(ArcadeGraph.java:133)
... 25 more
Caused by: com.arcadedb.utility.LockException: Database './arcadedb/databases/graph' is locked by another process (path=C:\[...]\.\arcadedb\databases\graph)
at com.arcadedb.database.EmbeddedDatabase.lockDatabase(EmbeddedDatabase.java:1628)
at com.arcadedb.database.EmbeddedDatabase.checkForRecovery(EmbeddedDatabase.java:1730)
at com.arcadedb.database.EmbeddedDatabase.openInternal(EmbeddedDatabase.java:1776)
... 30 more
Caused by: java.nio.channels.OverlappingFileLockException
at java.base/sun.nio.ch.FileLockTable.checkList(FileLockTable.java:229)
at java.base/sun.nio.ch.FileLockTable.add(FileLockTable.java:123)
at java.base/sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1321)
at java.base/java.nio.channels.FileChannel.tryLock(FileChannel.java:1191)
at com.arcadedb.database.EmbeddedDatabase.lockDatabase(EmbeddedDatabase.java:1607)
... 32 more
This second error also occurs when setting arcadedb.server.defaultDatabases, even if no databases were created beforehand.
There is a workaround for this issue: Replace the last (!) forward slash in the gremlin.arcadedb.directory property with a backslash when running the ArcadeDB Server on Windows, then everything will work just fine:
gremlin.arcadedb.directory=./arcadedb/databases\graph
However, this makes the configuration OS dependent and you cannot easily re-use the configurations without manually editing it for each installation. The database directory can also be represented in different ways (e.g. with absolute instead of relative file paths) which would also cause the same issue in the current implementation.
I already found a solution for this issue, I'm just not 100% sure if it covers all cases.