Proposer stuck due to block hash does not match the hash at the expected height

[antonio-altr]

Describe the bug
I have an OP stack running for several months, and both the batcher and proposer ran out of funds a month ago.
After topping up both accounts, the batcher is running fine again, but the proposer is unable to send txns to Sepolia due to this error:

t=2023-10-27T07:57:41+0000 lvl=info msg="Creating tx"                         service=proposer to=0x3B3f74F4385F31206E2482d0E7775a826B7c14b3 from=0x74a0a887f46927Aa8F20F947C795Da36984e5115
t=2023-10-27T07:57:41+0000 lvl=eror msg="Failed to send proposal transaction" err="failed to create the tx: failed to estimate gas: execution reverted: L2OutputOracle: block hash does not match the hash at the expected height" l1blocknum=4,417,789 l1blockhash=0xdd48feed5428ae9da110325dd01ec26c0b9ccf9e93af4d2428e40dc5246d477f l1head=4,572,075
t=2023-10-27T07:57:41+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,077 l1blocknum=4,572,078
t=2023-10-27T07:57:54+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,078 l1blocknum=4,572,078
t=2023-10-27T07:58:06+0000 lvl=info msg="Creating tx"                         service=proposer to=0x3B3f74F4385F31206E2482d0E7775a826B7c14b3 from=0x74a0a887f46927Aa8F20F947C795Da36984e5115
t=2023-10-27T07:58:06+0000 lvl=eror msg="Failed to send proposal transaction" err="failed to create the tx: failed to estimate gas: execution reverted: L2OutputOracle: block hash does not match the hash at the expected height" l1blocknum=4,417,789 l1blockhash=0xdd48feed5428ae9da110325dd01ec26c0b9ccf9e93af4d2428e40dc5246d477f l1head=4,572,077
t=2023-10-27T07:58:06+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,079 l1blocknum=4,572,080
t=2023-10-27T07:58:18+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,080 l1blocknum=4,572,080
t=2023-10-27T07:58:30+0000 lvl=info msg="Creating tx"                         service=proposer to=0x3B3f74F4385F31206E2482d0E7775a826B7c14b3 from=0x74a0a887f46927Aa8F20F947C795Da36984e5115
t=2023-10-27T07:58:30+0000 lvl=eror msg="Failed to send proposal transaction" err="failed to create the tx: failed to estimate gas: execution reverted: L2OutputOracle: block hash does not match the hash at the expected height" l1blocknum=4,417,789 l1blockhash=0xdd48feed5428ae9da110325dd01ec26c0b9ccf9e93af4d2428e40dc5246d477f l1head=4,572,079
t=2023-10-27T07:58:30+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,081 l1blocknum=4,572,082
t=2023-10-27T07:58:42+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,082 l1blocknum=4,572,082
t=2023-10-27T07:58:55+0000 lvl=info msg="Creating tx"                         service=proposer to=0x3B3f74F4385F31206E2482d0E7775a826B7c14b3 from=0x74a0a887f46927Aa8F20F947C795Da36984e5115
t=2023-10-27T07:58:55+0000 lvl=eror msg="Failed to send proposal transaction" err="failed to create the tx: failed to estimate gas: execution reverted: L2OutputOracle: block hash does not match the hash at the expected height" l1blocknum=4,417,789 l1blockhash=0xdd48feed5428ae9da110325dd01ec26c0b9ccf9e93af4d2428e40dc5246d477f l1head=4,572,081
t=2023-10-27T07:58:55+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,083 l1blocknum=4,572,084
t=2023-10-27T07:59:07+0000 lvl=dbug msg="waiting for l1 head > l1blocknum1+1" l1head=4,572,084 l1blocknum=4,572,084

This error seems to be generated from here:

optimism/packages/contracts-bedrock/src/L1/L2OutputOracle.sol

Line 210 in a37f81d

"L2OutputOracle: block hash does not match the hash at the expected height"

Based on the source code comments, it seems the way around this is to let the proposer send proposeL2Output without the _l1BlockHash value. May I know how that can be done, or whether there's a different way around this problem?

To Reproduce
Steps to reproduce the behavior:

1. Launch an OP stack according to the guide (https://stack.optimism.io/docs/build/getting-started/)
2. Let batcher and proposer run out of funds for a long interval, then top up both accounts
3. Confirm batcher is back to running normally
4. Confirm proposer proposeL2Output txns are rejected by the L2OutputOracle due to the error above

Additional context
Proposer: https://sepolia.etherscan.io/address/0x74a0a887f46927aa8f20f947c795da36984e5115
Batcher: https://sepolia.etherscan.io/address/0x2f5146dad1d8d30caa695774b6e233f5405c0823

[0xtrou]

Confirmed I had the same issue.
My chain was newly created, it cannot propose any new transactions.

[0xtrou]

cant find any ways to make proposer work again

[0xtrou]

did you manage to make it works again? @antonio-altr

[antonio-altr]

did you manage to make it works again? @antonio-altr

No, my proposer is still stuck here

[chunter-cb]

Same issue here

[opfocus]

This is usually because the op-node has not caught up with the L1 block, meaning it hasn't completed synchronization. You can check the synchronization status in the op-node log.

[antonio-altr]

@opfocus thank you, yes it seems after a long while the proposer is finally able to publish txns again, likely because of the op-node synchronization as you said.

[litus324]

@opfocus thank you, yes it seems after a long while the proposer is finally able to publish txns again, likely because of the op-node synchronization as you said.

can u tell me how did u fix it pls? i had the same problem and running it for 3day but it still not working. @antonio-altr @opfocus

[opfocus]

@opfocus thank you, yes it seems after a long while the proposer is finally able to publish txns again, likely because of the op-node synchronization as you said.

can u tell me how did u fix it pls? i had the same problem and running it for 3day but it still not working. @antonio-altr @opfocus

INFO [02-22|08:58:05.387] Sync progress reason="reconciled with L1"

this info from op-node log,. if it have not synced, it will display the l1 block height

[litus324]

@opfocus thank you, yes it seems after a long while the proposer is finally able to publish txns again, likely because of the op-node synchronization as you said.

can u tell me how did u fix it pls? i had the same problem and running it for 3day but it still not working. @antonio-altr @opfocus

INFO [02-22|08:58:05.387] Sync progress reason="reconciled with L1"

this info from op-node log,. if it have not synced, it will display the l1 block height

so can u tell me what should i do now. bc after running for a long time, my op-node and op-bathcer now is in a new status, and proposer is still not working :<

[opfocus]

Please check your op-geth logs, it seems that there is also a problem with op-geth

[askucher]

and?