Can't configure socket timeout for checkpoint sync

[wmitsuda]

lighthouse 3.2.1, macos

It seems there is no way to configure timeout for the checkpoint sync call, and from my tests it looks like it is hardcoded to 1 min.

Unfortunately, that makes checkpoint sync pretty unusable for mainnet, I'm trying various public endpoints, and they are much slower than 1 min (https://twitter.com/wmitsuda/status/1589857647217766400).

In fact, I'm trying to download a finalized state using curl right now from one public provider and it's already 9 min and counting!

I suggest a CLI flag to override this timeout, also increasing the default to like 20min should help.

[wmitsuda]

Ex: try the public endpoint from sigp

Nov 08 06:14:33.760 INFO Starting checkpoint sync                remote_url: https://mainnet.checkpoint.sigp.io/, service: beacon
Nov 08 06:15:36.928 CRIT Failed to start beacon node             reason: Error loading checkpoint state from remote 0x701af8e14a4407dbef993f21fdf52ac860f6a5b7af3e3cf3d9be5cbf037066ae: Reqwest(reqwest::Error { kind: Body, source: TimedOut })
Nov 08 06:15:36.928 INFO Internal shutdown received              reason: Failed to start beacon node

[michaelsproul]

Sorry yeah, there's an (immutable) timeout of 60 seconds since #3478 was implemented in #3521.

As a temporary workaround you can checkpoint sync with v3.1.2 and then switch to v3.2.1 (or muck around with manual checkpoint sync, but I know that's annoying).

[michaelsproul]

We should add a flag --checkpoint-sync-url-timeout which takes a value in seconds.

[wmitsuda]

ok, I think I was able to bootstrap the checkpoint sync with 3.1.0 (the hardcoded timeout was introduced in 3.1.2):

Nov 08 06:59:03.462 INFO Starting checkpoint sync                remote_url: https://mainnet.checkpoint.sigp.io/, service: beacon
Nov 08 07:00:48.309 INFO Loaded checkpoint block and state       state_root: 0x166f7be319fadd157a506f162a5a3b06822ed792c18b5a89a2a51bac41735471, block_root: 0x86943f5962c72e56c3868fd46de62de42e1d76fec6f0b9f2bf6041e3e0bb74f6, slot: 5088800, service: beacon

it took almost 2 min from logs, from my local machine (locate in South America) to sigp public endpoint.

[wmitsuda]

will wait for it to finish and then try to upgrade to 3.2.1, thanks for the help!

[michaelsproul]

as soon as the checkpoint is loaded it's safe to restart (you don't need to wait for e.g. block backfill)

[michaelsproul]

Resolved by #3710 🎉 Will be in our next release