feat: keep older vote messages for leaders in consensus pool

[akhi3030]

Problem

If we discard vote messages for any slots earlier than root slot, then the node that will be the leader in 8 slots will not be able to include them in the certs for rewards.

Summary of Changes

Similar to #513, this updates the consensus pool to keep older votes.

Part of #406

[qkniep]

why is this < here, whereas we use <= in BLSVerifier?

[akhi3030]

good catch, I wanted to ask the original authors about that as well.

Looking at: 308af20#diff-2077122be05c942393f87ed53352bb244607026554c88290d4d76b98c1124801R197 and 7e8e58b#diff-ea37759cc14ea84779ba69701cf8244e8314ad695cb412f76a4e3f103d57ffa0R129, it seems like @wen-coding is the author in both cases.

@wen-coding: do you know why we use < here and <= in BLSVerifier? I would've expected us to use <= in both cases.

[wen-coding]

I was worried that changing <= to < in BLSVerifier will block all the standstill messages related to root so we don't know how many others are agreeing with me. We don't use it for now, but maybe we want to output a warning "I'm in standstill and how many people are at the same slot with me".

[qkniep]

If you have finalized a block in slot s, even during standstill there is no relevant information gained by seeing a finalization from someone for slot s. Sending the finalization during standstill only matters for nodes that are actually behind to catch up to the latest finalized slot.

[AshwinSekar]

Some ideas I had about packing aggregates into the block footer:

• These s - 8 votes are not required for consensus, and we don't need to update or track any thresholds on these old slots
• Ideally we could avoid requiring the consensus pool be shared between multiple threads
• Instead what if we had a separate rewards container:
  • If sigverifier receives a vote > root send it to the ConsensusPool, if it's a notar or skip and we are leader for vote_slot + 8 also send it to the rewards container
  • If sigverifier receives a notar/skip vote where we are leader for vote_slot + 8 > root send it to rewards container
• This container doesn't have to do anything fancy, just store one vote per validator per slot
• When the block creation loop is about to finish producing the block, it can call the BLS aggregation fn for s - 8 and record the result in the block footer
• At this point we can clean up s - 8 from the rewards container

Separates the rewards tracking from the main consensus tracking so we don't have to mess with the ConsensusPool. Maybe wrap the VoteMessage in an Arc to avoid the overhead of sending it twice. Let me know what you think.

[akhi3030]

I like the idea, not polluting consensus to handle rewards makes sense to me!

[rogerANZA]

separate rewards container:

This was the original plan, to handle the rewards with a separate program. Somebody told us not to do that. In any case, I'm still a bit insecure about the hole one vote is enough for 100% rewards plan. I'm trying to cook something better.

[akhi3030]

separate rewards container:

This was the original plan, to handle the rewards with a separate program. Somebody told us not to do that. In any case, I'm still a bit insecure about the hole one vote is enough for 100% rewards plan. I'm trying to cook something better.

@rogerANZA : by separate program do you mean a separate smart contract? The proposal from Ashwin is to use a separate rust component inside anza.

In any case, I'm still a bit insecure about the hole one vote is enough for 100% rewards plan. I'm trying to cook something better.

Cool, let us know how you proceed.