Skip to content

Conversation

@jsvisa
Copy link
Contributor

@jsvisa jsvisa commented Aug 28, 2025

db inspect on the full database currently takes 30min+, because the db iterate was run in one thread, propose to split the key-space to 256 sub range, and assign them to the worker pool to speed up.

After the change, the time of running db inspect --workers 16 reduced to 10min(the keyspace is not evenly distributed).

jsvisa added 5 commits August 28, 2025 06:38
Signed-off-by: jsvisa <[email protected]>
Signed-off-by: jsvisa <[email protected]>
Signed-off-by: jsvisa <[email protected]>
Signed-off-by: jsvisa <[email protected]>
@jsvisa jsvisa requested a review from rjl493456442 as a code owner August 28, 2025 07:47
Signed-off-by: jsvisa <[email protected]>
Signed-off-by: jsvisa <[email protected]>
@rjl493456442
Copy link
Member

rjl493456442 commented Sep 1, 2025

Test it on my partial sync'd sepolia node, 3 minutes to traverse the key value store with 124GB data.

INFO [09-01|10:49:05.154] Inspecting database                      count=1,520,828,429 size=124.31GiB elapsed=3m4.001s
INFO [09-01|10:49:15.033] Opened ancient database                  database=/home/gary/hdd/sepolia-archive/ancient/state readonly=true

+-----------------------+-----------------------------+------------+------------+
|       DATABASE        |          CATEGORY           |    SIZE    |   ITEMS    |
+-----------------------+-----------------------------+------------+------------+
| Key-Value store       | Headers                     | 52.21 MiB  |      90001 |
| Key-Value store       | Bodies                      | 10.79 GiB  |      90001 |
| Key-Value store       | Receipt lists               | 3.38 GiB   |      90001 |
| Key-Value store       | Difficulties (deprecated)   | 0.00 B     |          0 |
| Key-Value store       | Block number->hash          | 3.60 MiB   |      90001 |
| Key-Value store       | Block hash->number          | 196.60 MiB |    5028001 |
| Key-Value store       | Transaction index           | 4.16 GiB   |  124066228 |
| Key-Value store       | Log index filter-map rows   | 0.00 B     |          0 |
| Key-Value store       | Log index last-block-of-map | 0.00 B     |          0 |
| Key-Value store       | Log index block-lv          | 0.00 B     |          0 |
| Key-Value store       | Log bloombits (deprecated)  | 0.00 B     |          0 |
| Key-Value store       | Contract codes              | 5.14 GiB   |     818148 |
| Key-Value store       | Hash trie nodes             | 0.00 B     |          0 |
| Key-Value store       | Path trie state lookups     | 175.93 MiB |    4499539 |
| Key-Value store       | Path trie account nodes     | 3.20 GiB   |   28212407 |
| Key-Value store       | Path trie storage nodes     | 23.95 GiB  |  238211659 |
| Key-Value store       | Path state history indexes  | 59.02 GiB  |  955373200 |
| Key-Value store       | Verkle trie nodes           | 0.00 B     |          0 |
| Key-Value store       | Verkle trie state lookups   | 0.00 B     |          0 |
| Key-Value store       | Trie preimages              | 61.92 MiB  |     882562 |
| Key-Value store       | Account snapshot            | 1.27 GiB   |   20498673 |
| Key-Value store       | Storage snapshot            | 12.63 GiB  |  175752705 |
| Key-Value store       | Beacon sync headers         | 2.08 GiB   |    3582837 |
| Key-Value store       | Clique snapshots            | 0.00 B     |          0 |
| Key-Value store       | Singleton metadata          | 2.35 KiB   |         13 |
| Ancient store (Chain) | Bodies                      | 105.98 GiB |    4938001 |
| Ancient store (Chain) | Receipts                    | 13.28 GiB  |    4938001 |
| Ancient store (Chain) | Headers                     | 1.85 GiB   |    4938001 |
| Ancient store (Chain) | Hashes                      | 178.95 MiB |    4938001 |
| Ancient store (State) | History.Meta                | 339.00 MiB |    4499538 |
| Ancient store (State) | Account.Index               | 5.03 GiB   |    4499538 |
| Ancient store (State) | Storage.Index               | 11.76 GiB  |    4499538 |
| Ancient store (State) | Account.Data                | 4.24 GiB   |    4499538 |
| Ancient store (State) | Storage.Data                | 1.98 GiB   |    4499538 |
+-----------------------+-----------------------------+------------+------------+
|                                    TOTAL            | 270.71 GIB | 1557285977 |

@rjl493456442 rjl493456442 changed the title cmd/geth: db inspect in parallel workers core/rawdb: inspect database in parallel Sep 1, 2025
@rjl493456442 rjl493456442 added this to the 1.16.3 milestone Sep 1, 2025
@rjl493456442 rjl493456442 merged commit 0cde527 into ethereum:master Sep 1, 2025
6 of 7 checks passed
gballet pushed a commit to gballet/go-ethereum that referenced this pull request Sep 11, 2025
`db inspect` on the full database currently takes **30min+**, because
the db iterate was run in one thread, propose to split the key-space to
256 sub range, and assign them to the worker pool to speed up.

After the change, the time of running `db inspect --workers 16` reduced
to **10min**(the keyspace is not evenly distributed).

---------

Signed-off-by: jsvisa <[email protected]>
Co-authored-by: Gary Rong <[email protected]>
@jsvisa jsvisa deleted the db-inspect-concurrent branch September 13, 2025 09:57
Sahil-4555 pushed a commit to Sahil-4555/go-ethereum that referenced this pull request Oct 12, 2025
`db inspect` on the full database currently takes **30min+**, because
the db iterate was run in one thread, propose to split the key-space to
256 sub range, and assign them to the worker pool to speed up.

After the change, the time of running `db inspect --workers 16` reduced
to **10min**(the keyspace is not evenly distributed).

---------

Signed-off-by: jsvisa <[email protected]>
Co-authored-by: Gary Rong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants