Stalled sync status is unclear

[eenagy]

Description

When on sepolia, and the client can't sync due to low peer count, the endpoint eth/v1/node/syncing returns false

Version

5.2.1 Linux, x64

Present Behaviour

1. Start el

 besu  --rpc-http-enabled --data-path=/home/eniko/.run-a-node/sepolia --network=SEPOLIA --engine-jwt-secret=$HOME/.run-a-node/sepolia/jwt.hex --rpc-http-api=ETH --rpc-http-port=8545

2. Start cl

lighthouse beacon_node  --disable-deposit-contract-sync --http --checkpoint-sync-url=https://beaconstate-sepolia.chainsafe.io --datadir=$HOME/.run-a-node/sepolia --execution-endpoint=http://localhost:8551 --execution-jwt=$HOME/.run-a-node/sepolia/jwt.hex --http-allow-origin="*" --http-port=5052 --network=sepolia

3. Check sync status

curl -s -X GET -H Content-Type: application/json http://localhost:5052/eth/v1/node/syncing
{"data":{"is_syncing":false,"is_optimistic":false,"el_offline":false,"head_slot":"5455360","sync_distance":"120"}}% 

curl -X GET "http://localhost:5052/lighthouse/syncing" -H  "accept: application/json" | jq
 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    18  100    18    0     0  21844      0 --:--:-- --:--:-- --:--:-- 18000
{
  "data": "Stalled"
}

Lighthouse log

Jul 17 06:56:03.336 INFO Logging to file                         path: "/home/eniko/.run-a-node/sepolia/beacon/logs/beacon.log"
Jul 17 06:56:03.341 INFO Lighthouse started                      version: Lighthouse/v5.2.1-9e12c21
Jul 17 06:56:03.341 INFO Configured for network                  name: sepolia
Jul 17 06:56:03.341 INFO Data directory initialised              datadir: /home/eniko/.run-a-node/sepolia
Jul 17 06:56:03.344 INFO Deposit contract                        address: 0x7f02c3e3c98b133055b8b348b2ac625669ed295d, deploy_block: 1273020
Jul 17 06:56:03.361 INFO Blob DB initialized                     oldest_blob_slot: Some(Slot(4243456)), path: "/home/eniko/.run-a-node/sepolia/beacon/blobs_db", service: freezer_db
Jul 17 06:56:03.368 INFO Starting checkpoint sync                remote_url: https://beaconstate-sepolia.chainsafe.io/, service: beacon
Jul 17 06:56:04.703 INFO Loaded checkpoint block and state       block_root: 0xc8a0e75e323cc959590d9dd21f27409cba1b8de51557b056387476fd1f8f474b, state_slot: 5455360, block_slot: 5455360, service: beacon
Jul 17 06:56:04.806 INFO Block production disabled               reason: no eth1 backend configured
Jul 17 06:56:04.916 INFO Beacon chain initialized                head_slot: 5455360, head_block: 0xc8a0…474b, head_state: 0xfa52…035c, service: beacon
Jul 17 06:56:04.916 INFO Timer service started                   service: node_timer
Jul 17 06:56:04.916 INFO UPnP Attempting to initialise routes
Jul 17 06:56:04.916 INFO Execution payloads are pruned           service: freezer_db
Jul 17 06:56:04.917 INFO ENR Initialised                         quic6: None, quic4: Some(9001), udp6: None, tcp6: None, tcp4: Some(9000), udp4: None, ip4: None, id: 0x51e7..6d19, seq: 1, enr: enr:-LW4QGXHQgD3DMv0MOVKXU2FEPS2HzWlRUKhPb9aSXl3-8hFT6Vf9I5wF4Q3EbnWBazXnwxfGFCcQtTUyvzlkBhHut4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDTH2GRkAAAc___________gmlkgnY0hHF1aWOCIymJc2VjcDI1NmsxoQPBDtLD1zT1tZzJAb2wBzVax9xPFkiHr7dyVsfJdjuZ-IhzeW5jbmV0cwCDdGNwgiMo, service: libp2p
Jul 17 06:56:04.918 INFO Libp2p Starting                         bandwidth_config: 3-Average, peer_id: 16Uiu2HAmRec9RjCJxtD37GUyZMZ6Ha7Pa2LU6B6mf7GE8t4ekEEB, service: libp2p
Jul 17 06:56:04.918 INFO Listening established                   address: /ip4/0.0.0.0/tcp/9000/p2p/16Uiu2HAmRec9RjCJxtD37GUyZMZ6Ha7Pa2LU6B6mf7GE8t4ekEEB, service: libp2p
Jul 17 06:56:04.918 INFO Listening established                   address: /ip4/0.0.0.0/udp/9001/quic-v1/p2p/16Uiu2HAmRec9RjCJxtD37GUyZMZ6Ha7Pa2LU6B6mf7GE8t4ekEEB, service: libp2p
Jul 17 06:56:04.918 INFO Deterministic long lived subnets enabled, subscription_duration_in_epochs: 256, subnets_per_node: 2, service: attestation_service
Jul 17 06:56:04.918 INFO Subscribing to long-lived subnets       subnets: [SubnetId(40), SubnetId(39)], service: attestation_service
Jul 17 06:56:04.919 INFO HTTP API started                        listen_address: 127.0.0.1:5052
Jul 17 06:56:05.024 INFO Execution engine online                 service: exec
Jul 17 06:56:05.024 INFO Issuing forkchoiceUpdated               forkchoice_state: ForkchoiceState { head_block_hash: 0x69d2c1fb4fdfff3bf8e9b197ee67be30709d29be6c6c9284b391c580171cbbea, safe_block_hash: 0x69d2c1fb4fdfff3bf8e9b197ee67be30709d29be6c6c9284b391c580171cbbea, finalized_block_hash: 0x69d2c1fb4fdfff3bf8e9b197ee67be30709d29be6c6c9284b391c580171cbbea }, service: exec
Jul 17 06:56:05.029 INFO Could not UPnP map Discovery port       error: Could not UPnP map port: 9000 on the gateway
Jul 17 06:56:05.040 INFO UPnP route established                  addr: /ip4/79.121.77.235/tcp/9000, service: libp2p
Jul 17 06:56:05.062 INFO UPnP route established                  addr: /ip4/79.121.77.235/udp/9001/quic-v1, service: libp2p
Jul 17 06:56:18.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Jul 17 06:56:18.001 INFO Searching for peers                     current_slot: 5455481, head_slot: 5455360, finalized_epoch: 170480, finalized_root: 0xc8a0…474b, peers: 0, service: slot_notifier
Jul 17 06:56:30.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Jul 17 06:56:30.001 INFO Searching for peers                     current_slot: 5455482, head_slot: 5455360, finalized_epoch: 170480, finalized_root: 0xc8a0…474b, peers: 0, service: slot_notifier
Jul 17 06:56:42.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Jul 17 06:56:42.001 INFO Searching for peers                     current_slot: 5455483, head_slot: 5455360, finalized_epoch: 170480, finalized_root: 0xc8a0…474b, peers: 0, service: slot_notifier
Jul 17 06:56:54.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Jul 17 06:56:54.001 INFO Searching for peers                     current_slot: 5455484, head_slot: 5455360, finalized_epoch: 170480, finalized_root: 0xc8a0…474b, peers: 0, service: slot_notifier
Jul 17 06:57:06.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Jul 17 06:57:06.001 INFO Searching for peers                     current_slot: 5455485, head_slot: 5455360, finalized_epoch: 170480, finalized_root: 0xc8a0…474b, peers: 0, service: slot_notifier
Jul 17 06:57:18.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Jul 17 06:57:18.001 INFO Searching for peers                     current_slot: 5455486, head_slot: 5455360, finalized_epoch: 170480, finalized_root: 0xc8a0…474b, peers: 0, service: slot_notifier
Jul 17 06:57:30.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Jul 17 06:57:30.001 INFO Searching for peers                     current_slot: 5455487, head_slot: 5455360, finalized_epoch: 170480, finalized_root: 0xc8a0…474b, peers: 0, service: slot_notifier

Expected Behaviour

I think it should return sync: true. While the Beacon API spec is unclear on this value, all other consensus clients return sync: true on stalled networks.

Steps to resolve

I would expect consistent behavior from all consensus clients when the peer count is 0.

[chong-he]

We have a fix coming in 5.3.0 to update the bootnodes: #6037

In the mean time, you can manually add them with the flag --boot-nodes

[eenagy]

We have a fix coming in 5.3.0 to update the bootnodes: #6037

In the mean time, you can manually add them with the flag --boot-nodes

Thank you. But I don't think that solves the issue with the stalled request having a sync status of false. I checked other clients; they also have peer issues, but they report syncing as true.

Also note, if I would test this against lighthouse logs, I would see sync started. So, there is some mismatch in how this is handled.

[michaelsproul]

Yeah that's a good point. This might have been a regression, I'll check it out

[chong-he]

We have a fix coming in 5.3.0 to update the bootnodes: #6037
In the mean time, you can manually add them with the flag --boot-nodes

Thank you. But I don't think that solves the issue with the stalled request having a sync status of false. I checked other clients; they also have peer issues, but they report syncing as true.

Also note, if I would test this against lighthouse logs, I would see sync started. So, there is some mismatch in how this is handled.

Sorry about that, I jumped to the conclusion too quickly once I see low peer count and Sepolia

Upon testing, the /lighthouse/syncing API is returning stalled as you mentioned, and the beacon API:

curl -X 'GET' 'http://localhost:5052/eth/v1/node/syncing' | jq is returning "is_syncing": false (which should be a true. perhaps because it registered as stalled currently in the Lighthouse API, hence it is returning a false here)

{
  "data": {
    "is_syncing": false,
    "is_optimistic": false,
    "el_offline": true,
    "head_slot": "5460800",
    "sync_distance": "133"
  }
}

[michaelsproul]

Resolved by:

• Return syncing on HTTP when sync is stalled #6129

Will be in 5.3