tilelive.all, SQLITE_CANTOPEN: with 1000+ .mbtiles files #90
Description
Hey all. I'm using tilelive.all (tilelive v. 4.5.3) to iterate over a folder full of unknown numbers of .mbtiles files (full of .pngs) and create Node routes to serve up the contents.
In certain environments, I'm able to call tilelive.all on a folder with 1089 .mbtiles files and all files to load as expected (my Macbook Pro, for example). On a small EC2 Ubuntu 14 instance, loading 400 files is fine, but if I add a few hundred more, then tilelive.all seems to log exactly 55 "SQLITE_CANTOPEN: unable to open database file" errors. It subsequently fires the callback for tilelive.all which attempts to tilelive.load each particular file. The error for each .load callback is also "SQLITE_CANTOPEN: unable to open database file".
The question is - is there a limit to the number of .mbtiles that can be opened by tilelive.all at a single time?
Here is a snippet:
//Load mbtiles from mbtiles folder.
tilelive.all(mbtilesLocation, function (err, list) {
//Create endpoint for mbtiles service, 1 for each source
list.forEach(function (file) {
var mbtilespath = 'mbtiles://' + path.join(mbtilesLocation, file.basename);
tilelive.load(mbtilespath, function (err, source) {
if (err)
throw err;
...
A few things I have tried:
-
Assuming that there was some limit to the number of SQLITE files that could be opened simultaneously, I added a setInterval with a delay of between .2 and 5 seconds to the tilelive.load operation to give it some time to breathe. (in hopes that a particular SQLITE file could be opened and closed/disposed of before moving on). This seemed to make a difference, sometimes.
-
The other main experiment has been to adjust the number of .mbtiles in the folder that gets loaded to see what effect this has. Just now, I started with 1089 files, which throws 55 CANTOPEN errors, and then subsequently fails to tilelive.load each individual file, throwing the same error message. I then removed 181 of these files - bringing the total number to 908. When I restart my Node app, tilelive.all doesn't log any of those CANTOPEN errors. I then begin to iterate over each file and tilelive.load them. Files node-get #1 thru Stream-list does not pass on info/metadata #125 load fine. On file Adjust number validation to exclude NaN #126 - copy: object validator callback #130, I get CANTOPEN errors in the tilelive.load callback. Files stream.deserialize buffers everything #131 - #800 load fine.
I then kill the app and restart. With the same number of .mbtiles files as just a minute ago, I now get no errors, and all files tilelive.load-ed correctly. Very inconsistent.
Any ideas about the limits of SQLITE/tilelive.all and why it might be behaving so inconsistently?
Thanks,
Ryan