feat: `engine_newPayloadV3`: validate, execute & store block #222

fmoletta · 2024-08-05T19:25:55Z

Motivation

Being able to fully validate execute and store blocks received by engine_newPayloadV3

Description

Add block validation, execution & storing to engine_newPayloadV3 endpoint

Fixes:

Genesis.get_block:
- Now uses the INITIAL_BASE_FEE as base_fee_per_gas
- Now computes roots instead of using zero as default
- Now uses default values for cancun fields instead of leaving them as None

(With these fixes the genesis' block hash now matches the parentBlockHash of the next block when running with kurtosis)

beacon_root_contract_call now sets the block's gas_limit to avoid tx validation errors

Misc:

compute_transactions_root is now a standalone function matching the other compute functions
Log more steps of the engine_newPayloadV3 endpoint
Improve api of ExecutionPayloadV3 & PayloadStatus

Other
We can now execute payloads when running with kurtosis 🚀

Disclaimer: We are still getting some execution errors in later blocks that we need to look into (They are all currently passing the block validations)

Closes #51

…re-cancun-time

…idation (#220) **Motivation** Fetch cancun time from DB when validating payload v3 timestamp  **Description** * Store cancun_time in db * Use the stored cancun_time when validating payload timestamp in `eth_newPayloadV3` * Replace update methods for chain data in `Store` with `set_chain_config` Bonus: * Move `NewPayloadV3Request` to its corresponding module and update is parsing to match the rest of the codebase   Closes None, but is part of #51

**Motivation** Having a way to obtain lates/earliest/pending/etc block numbers  **Description** * Add get and update methods for earliest, latest, finalized, safe & pending block number to `Store` & `StoreEngine` * Resolve block numbers from tag in rpc methods   Closes None, but fixes many and enables others

…pts & withdrawals (#225) **Motivation** These roots are currently being calculated using `from_sorted_iter` but without being sorted beforehand. This PR replaces this behavior with inserting directly into the trie to ensure that it is ordered, then computing the root (The same fix that has been previously applied to storage root) **Description** Fixes `compute_transactions_root`, `compute_receipts_root` & `compute_withdrawals_root`  **Notes** After this change, the payloads created by kurtosis local net now pass the block hash validations in `engine_NewPayloadV3`  Closes None, but is needed for #51 Co-authored-by: Federica Moletta <[email protected]>

…date-payload

**Motivation** Implement gas cost calculation updates defined in [EIP-7883](https://eips.ethereum.org/EIPS/eip-7883) Add boundary checks defined in [spec](https://github.com/ethereum/execution-specs/blob/51cabd86502df7af596f5c78a0c4e00a4f92822c/src/ethereum/osaka/vm/precompiled_contracts/modexp.py#L31)  **Description** * Adds gas cost calculation for modexp precompile as of Osaka fork * Adds boundary checks for modexp inputs as of Osaka fork * Gate base fee being charged if modules & base where == 0 under pre-osaka fork (Not in the spec & caused tests to fail) * `PrecompileFn` now takes a `fork: Fork` argument (there was no other way around this)   Closes lambdaclass#4155

…ss#4029) **Motivation** * Start the block building process as soon as we get a payload from a FCU * Only wait for the building process to finish when we get a `GetPayload` message * Keep latest 10 payloads in memory instead of filling up the DB with old payloads * Continuosly rebuild payload until requested/ slot time is up * Only remove transactions from mempool when we execute a payload  **Description** * Add `Blockchain` methods `initiate_payload_build` & `get_payload` to start & finish a payload build process * Add `Blockchain` field `payloads` containing a vector of payload ids and payload building tasks/ built payloads (max 10 at a time) * Remove any instance of payloads being stored in the DB   Closes lambdaclass#3920 --------- Co-authored-by: Tomás Grüner <[email protected]> Co-authored-by: Martin Paulucci <[email protected]>

**Motivation** I ran into the following panic while running ethrex on sepolia testnet: ``` thread 'tokio-runtime-worker' panicked at crates/networking/p2p/rlpx/eth/eth68/receipts.rs:30:69: index out of bounds: the len is 0 but the index is 0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ``` This PR prevents this panic from happening  **Description** * (eth68 message) `Receipts::new` no longer asumes we have at least one receipt for a given block   Closes #issue_number

**Motivation** In ARM machines, the implementation used always falls back to the default because AVX2 is x86-specific. **Description** Implement a NEON-specific version of blake2 (equivalent to the one for AVX2, but for NEON).  closes #4315

…es NEON..." (#4322) This reverts commit 97d4827. **Motivation**  **Description**  - This commit broke blockchain tests on ARM ```bash failures: blockchain_runner::istanbul/eip152_blake2/blake2/blake2b.json blockchain_runner::istanbul/eip152_blake2/blake2/blake2b_large_gas_limit.json blockchain_runner::istanbul/eip152_blake2/blake2/blake2b_gas_limit.json ```  Closes #issue_number

**Motivation** The server script runner doesn't stop if a run failed, restarting and not letting debug.  **Description** Now the script stops if a run failed   Closes #issue_number

…variable (#4328) **Motivation**  **Description**   Closes #issue_number

**Motivation**  **Description**  make replay test not required so we can merge despite it failing, maybe in the future when things become more stable we can make it required again.  Closes #issue_number

**Motivation** Current version just broadcast every transaction we receive to every active peer we have. This presents some issues on performance, memory and even in code responsibilities.  **Description** Reorganizes the transaction broadcast logic When a new peer connection is initialized, we send the hashes of all the txs in the mempool. Then every time we receive new txs, we add them to the mempool and send them to our peers, we send the txs to a sqrt of our peers, while sending only the hashes to the rest. The mempool itself is the one responsible for knowing which txs need to be broadcasted. Every time we add a new tx to the mempool we add it to the list of pending broadcast, then once we broadcast them, we erase them. Also if a tx is removed from the mempool before being broadcasted it is also deleted from the broadcast pool. There is one possible scenario where we could send a tx twice to the same peer, that is if after the tx left the mempool, we receive it again and add it back, we would send it again to the peers.   Closes #4241

**Motivation**  **Description** Adds an option to the makefile and server runner script to run with debug assertions enabled

**Motivation** The current update_pivot funciton in crates/networking/p2p/sync.rs has several room for improvement, primarily in the logic.  **Description** The function now estimates the new block number based on the block timestamp   Closes #4261 --------- Co-authored-by: Javier Rodríguez Chatruc <[email protected]>

**Motivation** Add the max blobs per tx check introduced by [EIP-7594](https://eips.ethereum.org/EIPS/eip-7594)  **Description** * Add max blobs per tx check to levm pre check sequence * Enable ef tests for eip-7594   Closes #4152

**Motivation**  **Description**  - ExecutionResult was inaccurate with LEVM, specially the gas used that subtracted the gas refunded but that's already done in LEVM. We don't have the ideal endpoints to try this out because the only ones that use this are `eth_estimateGas` and `eth_call`. The only way to see if it works fine is to try out the former RPC method with a transaction that triggers refunds... It would be good to have `eth_simulateV1` I guess Here's evidence that the gas_used in the VM is the `total gas used - refunded gas` https://github.com/lambdaclass/ethrex/blob/1918c5cedaf3f76df1bc01aba3b20d1d95162c84/crates/vm/levm/src/hooks/default_hook.rs#L199-L201 Bear with me, `exec_gas_consumed` is the same as `actual_gas_used` https://github.com/lambdaclass/ethrex/blob/1918c5cedaf3f76df1bc01aba3b20d1d95162c84/crates/vm/levm/src/hooks/default_hook.rs#L164  Closes #issue_number

**Motivation**  ETH client doesn't log too much, making it difficult to debug different problems. **Description**  Add trace, debug, and warning logs for better debugging.

**Motivation**  Sometimes we want to change the log level, primarly for debugging, and restarting the node would break the case of study. **Description**  Add a custom `admin_setLogLevel` endpoint that enables the node operator to specify a new log filter, just like with `RUST_LOG`. How to test: 1. Run a node (e.g. `ethrex --dev`) 2. Change the log levels: ``` curl localhost:8545 -H 'content-type: application/json' -d '{"jsonrpc": "2.0", "id": "1", "method": "admin_setLogLevel", "params": ["ethrex_dev::block_producer=info"]}' ``` 3. You should now only see `ethrex_dev::block_producer` logs > [!WARNING] > The `admin` namespace is currently unauthenticated and cannot be turned off. Be aware of this in public nodes  Closes #4299

**Motivation**  Currently the L2 metrics gatherer component scraps data every 1s, including L1 data (gas price and last committed/verified batch). This consumes a lot of unnecessary resources. **Description**  Increase the check interval to 5 seconds

**Motivation**  We lack a way to get the latest batch sealed, something similar to `eth_blockNumber`. **Description**  Add a new RPC method `ethrex_batchNumber` that returns the last sealed batch. How to test: 1. Start an L2 node: ``` COMPILE_CONTRACTS=true cargo run --release -- l2 --dev ``` 3. Wait about 1 minute until a batch is sealed. 4. Request the last batch number: ``` curl localhost:1729 -H 'content-type: application/json' -d '{"jsonrpc": "2.0", "id": "1", "method": "ethrex_batchNumber"}' ``` Should return something like `{"id":"1","jsonrpc":"2.0","result":"0x2"}`  --------- Co-authored-by: Copilot <[email protected]>

**Motivation**  We want to be able to compile ethrex without compiling revm, this PR enables it. **Description**   Closes #issue_number

**Motivation** Performance improvement on snap sync.  **Description** This PR refactors the P2P networking layer by centralizing peer scoring functionality into a unified system. The changes move from multiple scattered peer scoring implementations to a single `PeerScore` module that manages peer reputation and selection consistently across all sync operations. Key changes include: - Creation of a centralized `PeerScores` system to replace scattered scoring logic - Removal of local peer scoring implementations from state and storage healing modules - Updated peer selection to use the unified scoring system across all sync operations  Closes #2073 **Known issues** #4352: the usage of this method is required for scoring to be useful, but it shouldn't be used always for performance reasons. This is error prone, so eventually the scoring data should be embedded in the kademlia table's `PeerData`. --------- Co-authored-by: Javier Chatruc <[email protected]> Co-authored-by: Juan Munoz <[email protected]> Co-authored-by: Mateo Rico <[email protected]> Co-authored-by: Esteban Dimitroff Hodi <[email protected]> Co-authored-by: Lucas Fiegl <[email protected]> Co-authored-by: Gianbelinche <[email protected]> Co-authored-by: Francisco Xavier Gauna <[email protected]> Co-authored-by: ricomateo <[email protected]> Co-authored-by: juan518munoz <[email protected]> Co-authored-by: Javier Rodríguez Chatruc <[email protected]> Co-authored-by: Copilot <[email protected]>

**Motivation** Currenlty, we have no way of resuming an archive sync if it crashes/ we need to stop it. Our only option is to begin from the start, which can make us lose precious time in the case of unexpected crashes during long archive syncs. This PR proposes adding a `--checkpoint` flag which will periodically write the current archive sync's checkpoint data (current root, last hash, current file, etc) to a file. And which can later be used to resume archive sync from that same checkpoint  **Description**   Closes #issue_number --------- Co-authored-by: cdiielsi <[email protected]>

**Motivation** Remove cases where we using `as usize` to cast `u64` values when not needed  **Description**   Contributes to #4081

**Motivation** These handlers are currently sync and can block the runtime for quite a long time. This is not the ideal solution for this; we still need to do things better so the handler itself doesn't take a long time to fetch the requested nodes. We'll tackle this on a subsequent PR. **Description**   Closes #issue_number --------- Co-authored-by: Javier Chatruc <[email protected]>

**Motivation** **Description**   Closes #issue_number

) **Motivation**  Using "localhost" as default bind address can cause some problems. **Description**  Changed default to `0.0.0.0` for ETH RPC and `127.0.0.1` for EngineAPI

**Motivation** There are several possible sources of panic in `handshake.rs` and `codec.rs`  **Description** Solves the possible panics   Closes #4314

**Motivation** When adding support for trie iteration from a given starting point, we added a `debug_assert!` that checks that the internal stack isn't empty, but this is incorrect; it is entirely possible that the stack is empty if the trie is empty, in that case we simply return an empty iterator, which is fine. **Description**   Closes #issue_number

…EFTests (#4284) **Motivation** Implement changes defined in [EIP-7918](https://eips.ethereum.org/EIPS/eip-7918) & [EIP-7892](https://eips.ethereum.org/EIPS/eip-7892) Fix EFTests not using the `BlobSchedule` defined in the test files  **Description** * Update `calc_excess_blob_gas` according to EIP * Add `ForkBlobSchedule`s for Osaka & BPO1-BPO5 forks * Enable EIP-7918 & EIP-7892 ef tests * Use the `BlobSchedule` defined in each test file when running EF Tests   Closes #4157 & Closes #4156

…ckRangeUpdate` handling (#4360) **Motivation** Being able to support eth/69 messages  **Description** * Add missing validation to `BlockRangeUpdate` (latest_hash != zero) * Disconnect from peers if `BlockRangeUpdate` is invalid * Enable eth/69 as supported capability * Use negotiated eth capability version when decoding incoming messages * Enable hive tests for `BlockRangeUpdate`   Closes ##2785

…4363) **Motivation** The TxBroadcaster does not follow the spec, since it can send a tx to a peer that already knows of it  **Description** Adds a HashMap stating which transactions each peer knows, and avoiding sending a tx if a peer already knows it. After some time, this HashMap is pruned.   Closes #4356

**Motivation**  **Description**  Add a `docker-compose.yaml` file that deploys a Consensus client (lighthouse) and ethrex with just one command: ```sh docker compose up ``` `ETHREX_NETWORK` can be used to set the network, default is `mainnet`.  --------- Co-authored-by: Copilot <[email protected]>

fmoletta added 11 commits August 2, 2024 11:43

wip

2f4b506

Insert cancun time from genesis + update newpayloadv3 parsing

d08ee6b

fmt

14b8bae

Merge branch 'main' of https://github.com/lambdaclass/ethrex into sto…

f395676

…re-cancun-time

Update

507e969

Merge branch 'main' of https://github.com/lambdaclass/ethrex into sto…

a618b61

…re-cancun-time

replace update_x with set_chain_config method for StoreEngine

688fb6e

Merge branch 'main' of https://github.com/lambdaclass/ethrex into sto…

bbb4e52

…re-cancun-time

Add payload status methods + validate block header

e81c9cd

Fix

938ae60

Execute & store block

7c43792

fmoletta changed the base branch from main to store-cancun-time August 5, 2024 19:26

Store block number

00ab2e4

Base automatically changed from store-cancun-time to main August 6, 2024 14:11

Federica Moletta added 2 commits August 6, 2024 13:43

Remove unwrap

72c4d29

Fix root computation

5ce36ce

Federica Moletta and others added 11 commits August 7, 2024 15:10

Use base fee for block 0 on validation

2cea78c

Compute roots for genesis block

934a3fd

Set initial base fee for genesis block

b085177

Update default values for genesis block

904f86a

Revert change to block validation

38b2ef7

Fix gas limit for beacon root contract call

96f0da7

Add more tracing

abf5a8f

clippy

ee655a0

Update test

4b03b11

Merge branch 'main' of github.com:lambdaclass/ethereum_rust into vali…

45afb13

…date-payload

Add test

e39caf5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: `engine_newPayloadV3`: validate, execute & store block #222

feat: `engine_newPayloadV3`: validate, execute & store block #222

Uh oh!

fmoletta commented Aug 5, 2024 •

edited

Loading

Uh oh!

Uh oh!

feat: engine_newPayloadV3: validate, execute & store block #222

feat: engine_newPayloadV3: validate, execute & store block #222

Uh oh!

Conversation

fmoletta commented Aug 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

feat: `engine_newPayloadV3`: validate, execute & store block #222

feat: `engine_newPayloadV3`: validate, execute & store block #222

fmoletta commented Aug 5, 2024 •

edited

Loading