Database growth during non-finality

[michaelsproul]

Description

Lighthouse presently stores 1 state per epoch in the hot database during periods of non-finality. During a multi-week period of non-finality this would lead to hundreds of gigabytes of states being stored and the likely exhaustion of many nodes' disk storage.

Steps to resolve

@dapplion and I have discussed adapting the HDiff approach (from #5978) to the hot database, and think we have a fairly good scheme that goes as follows:

1. Store hdiffs in the hot database with state_root: Hash256 references rather than slot: Slot.
2. Retain the diff path from the most recent snapshot to the split state, while pruning everything else with a slot prior to the split slot.

[michaelsproul]

Fixed by:

• Hierarchical state diffs in hot DB #6750

🎉