Add a mode to run without freezer states

[michaelsproul]

Description

Presently Lighthouse copies finalized states to the freezer DB upon finalization. Validating nodes do not actually require these states at all and would be justified in dropping them entirely. Other clients (e.g. Teku and Nimbus) already implement this mode of operation.

I think it should be fairly straight-forward to implement, we could wrap the copying of states in a conditional, and use the anchor.state_upper_limit to inform the rest of the Lighthouse about the unavailability of the historic states. We'd want to maintain the invariant that anchor.state_upper_limit == split.slot. No schema changes would be required, because we'd essentially be maintaining a node that looks very similar to a recently checkpoint-synced node (albeit with full block history).

lighthouse/beacon_node/store/src/hot_cold_store.rs

Lines 1467 to 1484 in 6044984

	let mut cold_db_ops: Vec<KeyValueStoreOp> = Vec::new();
	if slot % store.config.slots_per_restore_point == 0 {
	let state: BeaconState<E> = get_full_state(&store.hot_db, &state_root, &store.spec)?
	.ok_or(HotColdDBError::MissingStateToFreeze(state_root))?;
	store.store_cold_state(&state_root, &state, &mut cold_db_ops)?;
	}
	// Store a pointer from this state root to its slot, so we can later reconstruct states
	// from their state root alone.
	let cold_state_summary = ColdStateSummary { slot };
	let op = cold_state_summary.as_kv_store_op(state_root);
	cold_db_ops.push(op);
	// There are data dependencies between calls to `store_cold_state()` that prevent us from
	// doing one big call to `store.cold_db.do_atomically()` at end of the loop.
	store.cold_db.do_atomically(cold_db_ops)?;

Nodes without any historic states prior to the most recent finalized state would frustrate current Lighthouse's ability to checkpoint sync, so I think we should also implement #3210 around the same time.

[michaelsproul]

I now think the best way to achieve this might be to set state_upper_limit to u64::MAX when historic state pruning is enabled, i.e. some flag like --prune-historic-states true is set.

[michaelsproul]

I'm working on this currently