Reorg handling is beyond slow

[pokrovskyy]

Due to quick block times, reorgs happen rather often on Fractal Bitcoin, and it seems there is an issue with efficient reorg handling. I'm running ord version with redb-2.1.1 (before recent redb-2.0.0 downgrade). My Fractal Mainnet index is around 380GB and it takes around 5 hrs to roll back the db upon reorg (testnet is around 92gb and takes around 1 hr to roll back which seems proportional to index size). I am running datacenter-grade server with 2TB NVMe RAID and 64GB RAM. See reorg-related logs for just the last day:

[2024-11-18T22:41:41Z INFO  ord::index::reorg] rolling back database after reorg of depth 2 at height 205611
[2024-11-19T03:40:07Z INFO  ord::index::reorg] successfully rolled back database to height 205600

<rollback took 5 hrs, worked for 2 hrs until next reorg>

[2024-11-19T05:48:25Z INFO  ord::index::reorg] rolling back database after reorg of depth 2 at height 206429
[2024-11-19T11:18:07Z INFO  ord::index::reorg] successfully rolled back database to height 206400

<rollback took 5.5 hrs, worked for 1 hr until next reorg>

[2024-11-19T12:21:27Z INFO  ord::index::reorg] rolling back database after reorg of depth 2 at height 207196
[2024-11-19T17:16:54Z INFO  ord::index::reorg] successfully rolled back database to height 207180

<rollback took 5 hrs, worked for 2 hrs until next reorg>

[2024-11-19T19:05:13Z INFO  ord::index::reorg] rolling back database after reorg of depth 2 at height 207953
[2024-11-19T23:46:38Z INFO  ord::index::reorg] successfully rolled back database to height 207940

As you can see, in my case ord spends most of the time rolling back the index and only a few hrs between reorgs. Anyone experiencing similar issues? Any special config to handle this better?

I haven't really dug into reorg handling code yet, but it seems it makes checkpoints to restore to in case of a reorg. I only do not understand why is it so slow? Surely it doesn't need to rewrite the whole DB, I would imagine it only needs to roll back to previous checkpoint, so idk why does it take 5 hrs to do so...

I am now re-syncing with redb-2.0.0 downgrade, will see if it makes any difference, but any feedback is welcome!