Attestation simulator errors on attestations that don't match source

[michaelsproul]

Description

For a short time after startup the attestation simulator logs some errors like:

Jan 09 00:45:24.613 ERRO Failed to get attestation participation flag indices, unaggregated_attestation: Attestation { aggregation_bits: Bitfield { bytes: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], len: 438, _phantom: PhantomData<ssz_types::bitfield::Variable<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UTerm, typenum::bit::B1>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>>> }, data: AttestationData { slot: Slot(8161408), index: 0, beacon_block_root: 0x3279754d2d97565765778a54a0cfe6251bff3e393454ca692a933693890bba30, source: Checkpoint { epoch: Epoch(255042), root: 0x9c5fd35fcc6a03f206f72819425168408f7c4ea4044c97c5a1f9a528089c8ddf }, target: Checkpoint { epoch: Epoch(255044), root: 0x3279754d2d97565765778a54a0cfe6251bff3e393454ca692a933693890bba30 } }, signature: 0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 }, error: IncorrectAttestationSource, service: val_mon, service: beacon

The log immediately prior to that is:

Jan 09 00:45:24.455 INFO New block received root: 0x139323bced87e7af856d42888c94b441c53a31801ebd5f72ab2d1259fecfad73, slot: 8161425

It seems like the node produces a bunch of low-quality attestations after starting up/checkpoint syncing for the first time, while it's syncing. These attestations are then reported on a slight delay (e.g. the attestation from slot 8161408 is reported at slot 8161425, ~17 slots later).

Version

v4.5.0 unstable at b47e3f2

Steps to resolve

Option 1: Handle the error

We could handle the IncorrectAttestationSource error and convert it to a source_hit=false, target_hit=false, head_hit=false case. The downside of this is that it hurts the attestation simulator metrics/stats, although arguably a node that's offline/syncing and producing these low-quality attestations should have its metrics reflect that.

Option 2: Don't simulate attestations while syncing

Alternatively if we want to avoid updating the metrics with these attestations without logging errors, we could update the attestation simulator to avoid producing attestations while syncing.

I'm currently leaning towards option 1.

Additional Info

Attestation simulator added in #4880.