Memory issues during genesis sync #3730
Description
Description
MigaLabs and now Gnosis have both reported Lighthouse using a lot of memory during genesis sync. We thought it might have been #2944, but with that issue resolved the problem is still occurring.
Looking at some metrics on the affected Gnosis servers it seems not to be a memory fragmentation issue, as the non-mmap memory is high (~10GB) and the free block memory is low (<800MB). I think to get to the bottom of this we would need to run lighthouse under heaptrack while syncing from genesis on mainnet or Gnosis chain.
Version
v3.2.1
Workarounds
Use checkpoint sync. It's a lot faster in multiple ways and unaffected by this issue.
There will be some checkpoint sync servers set up for Gnosis chain before the Gnosis merge.