Replies: 15 comments
-
|
Shure, that would be a bug. I did a quick profile as a baseline (using one of our test fixture and after this with 6x copies of a generated mbtiles of bavara (~500MB) via planetiler). Everything seems ordinary (fixture well below 1MB peak, 6x bavaria ~8MB peak) on my end. >250MB memory usage without doing anything seems uncharacteristic. flat does not tell me a lot about what is in the container. It just tells me that there are no duplciates:
Martins memory consumptioncan be profiled via Could you please report the following items?
If the data is NOT open/sharable, I would also like to have martin run with |
Beta Was this translation helpful? Give feedback.
-
|
Btw, a few notes on the options used:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the response.
The tiles are public, part of those generated by martin (MVT) in https://github.com/hiddewie/OpenRailwayMap-vector. I published the image to https://hub.docker.com/repository/docker/hiddewie/openrailwaymap-tiles/tags/latest/sha256:4e651901ba6f847bab642a9546367b3e03bc8ed77aceb6b823901ba936dbbd0e which should be public for everyone to download. With Docker Compose, start it with services:
martin-static:
image: hiddewie/openrailwaymap-tiles:latest
mem_limit: 250M
build:
dockerfile: martin-static.Dockerfile
ports:
- '3000:3000'You can extract the tile files with e.g. docker cp openrailwaymap-cartocss-martin-static-1:/tiles/high.mbtiles tiles/high.mbtiles(container name might be somewhat different for you)
Thanks, I will also try this locally to see if I can find anything of interesting that causes the memory consumption.
The format is MVT tiles. A recent generation can be seen in https://github.com/hiddewie/OpenRailwayMap-vector/actions/runs/13980023456/job/39143024570#step:6:61. There are many jobs, all generating tiles for a different region in parallel. After all tiles are generated for a region, they are merged together with A summary of the merged tiles can be found in e.g. https://github.com/hiddewie/OpenRailwayMap-vector/actions/runs/13980023456/job/39144919530#step:7:419 (one per source type). example meta output
Indeed, I get the comment, I was just trying everything I could think of to reduce memory usage, and see what works :) |
Beta Was this translation helpful? Give feedback.
-
|
So there are a few things that could "grow" internally besides the martin-controlled cache:
I can't think of anything else tbh... |
Beta Was this translation helpful? Give feedback.
-
|
On my host system |
Beta Was this translation helpful? Give feedback.
-
|
Some information from my kernel invoking the OOM killer. It seems to be one of the sqlite worker threads. Extract from |
Beta Was this translation helpful? Give feedback.
-
|
As a workaround, I am also investigating converting the MBTile files in to PMTiles (using Conversion with (Fish shell) for t in electrification high signals speed standard; pmtiles convert $t.{mbtiles,pmtiles}; endls -lah *.*tiles
-rw-r--r-- 1 hidde hidde 6.2M Mar 20 21:37 electrification.mbtiles
-rw-rw-r-- 1 hidde hidde 4.7M Mar 22 12:57 electrification.pmtiles
-rw-r--r-- 1 hidde hidde 913M Mar 20 21:34 high.mbtiles
-rw-rw-r-- 1 hidde hidde 824M Mar 22 12:58 high.pmtiles
-rw-r--r-- 1 hidde hidde 101M Mar 20 21:37 signals.mbtiles
-rw-rw-r-- 1 hidde hidde 89M Mar 22 12:58 signals.pmtiles
-rw-r--r-- 1 hidde hidde 32M Mar 20 21:36 speed.mbtiles
-rw-rw-r-- 1 hidde hidde 26M Mar 22 12:58 speed.pmtiles
-rw-r--r-- 1 hidde hidde 277M Mar 20 21:36 standard.mbtiles
-rw-rw-r-- 1 hidde hidde 230M Mar 22 12:58 standard.pmtilesResults look promising. If I then start martin with static configuration with the pmtiles only as tile sources: #mbtiles:
# sources:
# electrification: /tiles/electrification.mbtiles
# high: /tiles/high.mbtiles
# signals: /tiles/signals.mbtiles
# speed: /tiles/speed.mbtiles
pmtiles:
sources:
electrification: /tiles/electrification.pmtiles
high: /tiles/high.pmtiles
signals: /tiles/signals.pmtiles
speed: /tiles/speed.pmtiles
sprites: /symbols
fonts: /config/fontsThe container starts and runs with < 8MB memory, like expected |
Beta Was this translation helpful? Give feedback.
-
|
@nyurik I don't know why sqlite takes more memory in the container.. seems highly sus. |
Beta Was this translation helpful? Give feedback.
-
|
I am suspecting a problem that is a combination of Linux kernel, Docker and SQLite library in Martin. My kernel My Docker (I tried with both versions 27.3 and 28.0): |
Beta Was this translation helpful? Give feedback.
-
|
@hiddewie I think I have found a clue Edit: the 512MB is a mistake, forgot to carry the 8. Its 64Mebibyte, assuming B is a Bit. |
Beta Was this translation helpful? Give feedback.
-
|
512MiB is also our default.. |
Beta Was this translation helpful? Give feedback.
-
|
your config is correct as far as I am seing it here I have no clue why this defaults to 512, do you (@hiddewie) have a clue? Am I maybe just looking at the wrong file? In any case, will look at this further in the evening/tomorrow, need to do some excercise while the wearther is nice |
Beta Was this translation helpful? Give feedback.
-
|
@hiddewie consider telling martin about the config file, or the limited workers, or the cache-size ^^
|
Beta Was this translation helpful? Give feedback.
-
|
On the other hand, #1758 (comment) does have different (correct) output.. |
Beta Was this translation helpful? Give feedback.
-
|
@CommanderStorm re the 512MB cache If you look at the https://github.com/hiddewie/OpenRailwayMap-vector/blob/master/docker-compose.yml repository, there are actually two Martin services / configurations defined:
The things I tried to reproduce this issue (setting cache to 0 or 1 MB, setting workers to 1, pool size to 1), I all did by editing the command https://github.com/hiddewie/OpenRailwayMap-vector/blob/5ddfaa3bb24f0d07c273f3df7c7f6347c1670cf4/martin-static.Dockerfile#L7 to start the (You can find some of the testing I did in a temporary branch: https://github.com/hiddewie/OpenRailwayMap-vector/blob/7183059236d022533b7802ac0f131a66756b74f5/martin-static.Dockerfile. Be careful, it is a mess). |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Scenario:
Starting the container:
and the container shows exit reason "OOMKilled: true".
Giving the container 1000MB of memory, it starts and runs as usual:
Tested Martin v0.12.0 until v0.15.0
I tried:
--auto-bounds skip--workers 1--pool-size 1--cache-size 1During startup, the container still goes out of memory. It is unclear to me what it is doing, and how to further reduce memory usage.
Thanks for your consideration and ideas.
Beta Was this translation helpful? Give feedback.
All reactions