Expose additional resource utilization metrics on /metrics endpoint

[jsgranados]

Overview

As a node-runner, I would like to have additional resource utilization metrics exposed on the /metrics endpoint, so that I am able to better track the resource utilization that Algod uses and establish alerts on them

Details

• Metrics to expose:
  • process_open_fds (to track open file descriptors)
  • process_max_fds (to track the max file descriptors available)
  • process_cpu_seconds_total (to track total user and system CPU time spent in seconds, type: counter)
  • process_resident_memory_bytes (to track resident memory size in bytes, type: gauge)

Example Cases:

• Alerting on approaching open file limits
  CI: set -coverpkg when calling go test in nightly builds #3185

Potential Approach

• Add func newProcessCollector to our own algod implementation such that the /metrics endpoint can report algod's own process metrics
  • https://pkg.go.dev/github.com/prometheus/client_golang/prometheus/collectors#NewProcessCollector

[jsgranados]

Are we exposing these metrics already?