Add service verification to reachability check

[pschork]

Current reachability check only confirms that the port is open.

This new logic performs additional validation using grpc reflection to ensure we do not mislead operators when there are configuration issues preventing us from connecting to the node.

If port is open, we now verify the expected service is available based on port specification using grpc reflection. This will catch situations where port is open, but service backend is configured wrong or unreachable.

We now include a status message for dispersal and retrieval checks to help operators debug any reachability check failures.

all services available

{
  "operator_id": "39638352822b2eecf33f2ea0e6893a576edbcc202b4925c3ab2875677945bf58",
  "dispersal_socket": "98.84.160.141:32007",
  "retrieval_socket": "98.84.160.141:32008",
  "dispersal_online": true,
  "retrieval_online": true,
  "dispersal_status": "node.Dispersal is available",
  "retrieval_status": "node.Retrieval is available"
}

ports closed or unreachable

{
  "operator_id": "39638352822b2eecf33f2ea0e6893a576edbcc202b4925c3ab2875677945bf58",
  "dispersal_socket": "98.84.160.141:32002",
  "retrieval_socket": "98.84.160.141:32001",
  "dispersal_online": false,
  "retrieval_online": false,
  "dispersal_status": "port closed or unreachable for 98.84.160.141:32002",
  "retrieval_status": "port closed or unreachable for 98.84.160.141:32001"
}

port open but grpc fails to connect

previously this would return dispersal_online: true

{
  "operator_id": "39638352822b2eecf33f2ea0e6893a576edbcc202b4925c3ab2875677945bf58",
  "dispersal_socket": "98.84.160.141:32001",
  "retrieval_socket": "98.84.160.141:32008",
  "dispersal_online": false,
  "retrieval_online": true,
  "dispersal_status": "rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded",
  "retrieval_status": "available"
}

ports open but serving wrong grpc services

previously this would return dispersal_online: true and retrieval_online: true

{
  "operator_id": "39638352822b2eecf33f2ea0e6893a576edbcc202b4925c3ab2875677945bf58",
  "dispersal_socket": "98.84.160.141:32007",
  "retrieval_socket": "98.84.160.141:32008",
  "dispersal_online": false,
  "retrieval_online": false,
  "dispersal_status": "grpc available but node.Dispersal service not found at 98.84.160.141:32007",
  "retrieval_status": "grpc available but node.Retrieval service not found at 98.84.160.141:32008"
}

Checks

• I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
• I've checked the new test coverage and the coverage percentage didn't drop.
• Testing Strategy
  • Unit tests
  • Integration tests
  • This PR is not tested :(

[ian-shim]

lgtm, can you help with integrating v2 dispersal port into reachability check?