validator version 1.16.16 crashed with "unable to freeze a block with bankhash"

[bji]

Problem

I am running a primary/secondary validator setup with two physically separate validators. Both validators were running 1.16.16-jito w/local vote mods (the vote mods are not believed to alter any aspect of accounts state, because the mods only alter which slots are voted on in replay_stage.rs).

Both primary and secondary crashed simultaneously with the same error message. This is two separate validators experiencing the same issue. Note that the secondary likely took the tower from the primary shortly before the primary crashed as is common behavior when a secondary's tower gets out of sync with the primary.

The log message on both validators was:

[2023-10-16T22:47:33.894214958Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solReplayStage" one=1i message="panicked at 'We have tried to repair duplicate slot: 224116920 more than 10 times and are unable to freeze a block with bankhash 38rFNeVT1FGHResoXsgTN2cyyjBiJd6ZFcXWCPnXzZn7, instead we have a block with bankhash Some(GVZoNMHn1o8CB76i3LLRHjTA1p8f4iyTa6hn9eopWJh1). This is most likely a bug in the runtime. At this point manual intervention is needed to make progress. Exiting', core/src/replay_stage.rs:1585:25" location="core/src/replay_stage.rs:1585:25" version="1.16.16 (src:00000000; feat:4033350765, client:JitoLabs)"

I have collected the following files from one of the validators:

snapshot:
https://s3.us-west-1.amazonaws.com/shinobi-systems.com/incident-2023.10.16/snapshot-224107039-GTo6rcFciwNew8Bx9Hggv7n9oS5AZVeQGSXQMkaFfkfy.tar.zst

incremental snapshot:
https://s3.us-west-1.amazonaws.com/shinobi-systems.com/incident-2023.10.16/incremental-snapshot-224107039-224116213-qg6B13UGAAMKDm6XqG6p6PchnZ7RAgjhbe759igiFce.tar.zst

ledger (copied 10,000 slots leading up to the crash):
https://s3.us-west-1.amazonaws.com/shinobi-systems.com/incident-2023.10.16/ledger.tar.gz

validator logs leading up to the crash:
https://s3.us-west-1.amazonaws.com/shinobi-systems.com/incident-2023.10.16/validator.log.gz

[steviez]

@bji - Assuming you haven't completely wiped your ledger, can you please post the file that was created within <LEDGER_DIR>/bank_hash_details/. If you have multiple (ie if your node auto restarts on failure), please post the lowest slot.

If you have it, I'll replay the slot myself to generate a hopefully good file, and then compare mine vs yours to figure out what account(s) / transaction(s) might have caused the issue

[bji]

Happened again today, both primary and secondary failing with the same error:

thread 'solReplayStage' panicked at 'We have tried to repair duplicate slot: 229873482 more than 10 times and are unable to freeze a block with bankhash 51idF9qg8ntNMzhmUaMTvygV2nrPNCJ11uqQPvLsfVYU, instead we have a block with bankhash Some(ASJhPCw4LYXouujQSMZSMVLcfSU8dVwYTfv2czncF6Fv). This is most likely a bug in the runtime. At this point manual intervention is needed to make progress. Exiting', core/src/replay_stage.rs:1587:25
[2023-11-14T01:48:20.839284352Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solReplayStage" one=1i message="panicked at 'We have tried to repair duplicate slot: 229873482 more than 10 times and are unable to freeze a block with bankhash 51idF9qg8ntNMzhmUaMTvygV2nrPNCJ11uqQPvLsfVYU, instead we have a block with bankhash Some(ASJhPCw4LYXouujQSMZSMVLcfSU8dVwYTfv2czncF6Fv). This is most likely a bug in the runtime. At this point manual intervention is needed to make progress. Exiting', core/src/replay_stage.rs:1587:25" location="core/src/replay_stage.rs:1587:25" version="1.16.18 (src:00000000; feat:4033350765, client:JitoLabs)"

[bji]

@bji - Assuming you haven't completely wiped your ledger, can you please post the file that was created within <LEDGER_DIR>/bank_hash_details/. If you have multiple (ie if your node auto restarts on failure), please post the lowest slot.

If you have it, I'll replay the slot myself to generate a hopefully good file, and then compare mine vs yours to figure out what account(s) / transaction(s) might have caused the issue

I provided the entire ledger in the bug description. And the original ledger is long since gone, it is necessary to completely delete it in order for the validator to start successfully in this situation.

[steviez]

Assuming you haven't completely wiped your ledger, can you please post the file that was created within <LEDGER_DIR>/bank_hash_details/. If you have multiple (ie if your node auto restarts on failure), please post the lowest slot.

If you have it, I'll replay the slot myself to generate a hopefully good file, and then compare mine vs yours to figure out what account(s) / transaction(s) might have caused the issue

For completeness, just realized I misremembered and this feature did not get backported to v1.16 - sorry for the wild goose chase

[bji]

Assuming you haven't completely wiped your ledger, can you please post the file that was created within <LEDGER_DIR>/bank_hash_details/. If you have multiple (ie if your node auto restarts on failure), please post the lowest slot.
If you have it, I'll replay the slot myself to generate a hopefully good file, and then compare mine vs yours to figure out what account(s) / transaction(s) might have caused the issue

For completeness, just realized I misremembered and this feature did not get backported to v1.16 - sorry for the wild goose chase

No problem - but is the complete ledger I linked to in the bug description not sufficient? Erm I guess it's 10,000 copied slots, not the whole ledger. Is it enough?

[steviez]

No problem - but is the complete ledger I linked to in the bug description not sufficient?

The ledger you provided is sufficient for me to download and replay. However, replaying is not guaranteed to reproduce the failure (ie the majority of the cluster voted otherwise). That file I requested dumps the account state of every written account for that slot. So, even if we can't reproduce the failure, this file would allow me to see which account differed, and exactly how it differed. This is typically enough to work backwards to a specific TX, or we can see if it is a single bit flip or something

[bji]

No problem - but is the complete ledger I linked to in the bug description not sufficient?

The ledger you provided is sufficient for me to download and replay. However, replaying is not guaranteed to reproduce the failure (ie the majority of the cluster voted otherwise). That file I requested dumps the account state of every written account for that slot. So, even if we can't reproduce the failure, this file would allow me to see which account differed, and exactly how it differed. This is typically enough to work backwards to a specific TX, or we can see if it is a single bit flip or something

By single bit flip do you mean, a hardware error on the machine?

This is impossible in this particular instance because two machines had the same error at the same time. There is definitely something that actually happened to create consensus deviation.

[steviez]

By single bit flip do you mean, a hardware error on the machine?
This is impossible in this particular instance because two machines had the same error at the same time. There is definitely something that actually happened to create consensus deviation.

Sure, I'm not trying to brush off your report as a hardware fault. Whether it is a single bit or a whole 10 MB account, the file from when your node failed + a regenerated file from correct execution allow me to diff the account state. You may recall I had similar files when we had to momentarily back out one of your PRs, and those files seemingly helped us hone in on the issue much faster (I guess you could tell me since you did the in depth debugging 😉 )

Without this extra level detail, we are at the mercy of being able to reproduce the error reliably. But, given that the rest of the network deviated from how your node voted, I'm not optimistic we'd be able to reproduce. I'm pretty sure I pulled down those slots and attempted to reproduce a few weeks back, but I'm honestly not positive so I'll give it another go to be sure.

And to confirm - are you still running v1.16.16-jito ?

[bji]

By single bit flip do you mean, a hardware error on the machine?
This is impossible in this particular instance because two machines had the same error at the same time. There is definitely something that actually happened to create consensus deviation.

Sure, I'm not trying to brush off your report as a hardware fault. Whether it is a single bit or a whole 10 MB account, the file from when your node failed + a regenerated file from correct execution allow me to diff the account state. You may recall I had similar files when we had to momentarily back out one of your PRs, and those files seemingly helped us hone in on the issue much faster (I guess you could tell me since you did the in depth debugging 😉 )

Without this extra level detail, we are at the mercy of being able to reproduce the error reliably. But, given that the rest of the network deviated from how your node voted, I'm not optimistic we'd be able to reproduce. I'm pretty sure I pulled down those slots and attempted to reproduce a few weeks back, but I'm honestly not positive so I'll give it another go to be sure.

And to confirm - are you still running v1.16.16-jito ?

Sorry, I didn't mean to imply that you were trying to brush off the report. I just wanted to make sure that it was clear that this one is very unlikely to be a hardware fault.

Not still running 1.16.16-jito. The most recent occurance - a few hours ago - was on 1.16.18-jito.

If you don't have time to do the state diff, I could do it with some hinting on the tooling I'll need to accomplish this task. Don't mean to put this issue all on you!

[steviez]

Sorry, I didn't mean to imply that you were trying to brush off the report. I just wanted to make sure that it was clear that this one is very unlikely to be a hardware fault.

👍

Not still running 1.16.16-jito. The most recent occurance - a few hours ago - was on 1.16.18-jito.

👍

If you don't have time to do the state diff, I could do it with some hinting on the tooling I'll need to accomplish this task. Don't mean to put this issue all on you!

Ha, well that comment bring us full circle to my earlier comment:
"just realized I misremembered and this feature did not get backported to v1.16"

So, whether there is any additional work to do boils down to whether the bad hash can be created with the ledger you provided. As mentioned, I thought I tried but not positive so I'll pull these down and give it a run

[bji]

Just happened again. I'll save the full ledger.

[steviez]

For context, the original error:

We have tried to repair duplicate slot: 224116920 more than 10 times and are unable to freeze a block with 
bankhash 38rFNeVT1FGHResoXsgTN2cyyjBiJd6ZFcXWCPnXzZn7, instead we have a block with
bankhash Some(GVZoNMHn1o8CB76i3LLRHjTA1p8f4iyTa6hn9eopWJh1)

I downloaded your ledger and replayed with several versions, namely:

v1.16.16
v1.16.16-jito
v1.16.16-jito w/ jemalloc debug enabled (this has caught one or two bugs in the past)

All of these runs produced the correct hash 38rFNeVT1FGHResoXsgTN2cyyjBiJd6ZFcXWCPnXzZn7 on slot 224116920.

Given that you have seemingly been able to reproduce this problem several times, I would advise cherry-picking the commit that #32632 introduced onto whichever v1.16 branch you're running. If it fails again, it will produce a file in <LEDGER_DIR>/bank_hash_details that will be useful for us to track down the issue

[bji]

For context, the original error:

We have tried to repair duplicate slot: 224116920 more than 10 times and are unable to freeze a block with 
bankhash 38rFNeVT1FGHResoXsgTN2cyyjBiJd6ZFcXWCPnXzZn7, instead we have a block with
bankhash Some(GVZoNMHn1o8CB76i3LLRHjTA1p8f4iyTa6hn9eopWJh1)

I downloaded your ledger and replayed with several versions, namely:

v1.16.16
v1.16.16-jito
v1.16.16-jito w/ jemalloc debug enabled (this has caught one or two bugs in the past)

All of these runs produced the correct hash 38rFNeVT1FGHResoXsgTN2cyyjBiJd6ZFcXWCPnXzZn7 on slot 224116920.

Given that you have seemingly been able to reproduce this problem several times, I would advise cherry-picking the commit that #32632 introduced onto whichever v1.16 branch you're running. If it fails again, it will produce a file in <LEDGER_DIR>/bank_hash_details that will be useful for us to track down the issue

Thanks, much appreciated. Is it possible that there was something restricted just to an in-memory issue that never made it to the ledger itself?

Anyway, I will cherry pick that change and run it from now on. Will let you know if this happens again.