Add metrics related to leader

[jrhee17]

Sometimes we notice that replica.has.leadership reports 1 for more than one central dogma instance within a cluster.

ref:

centraldogma/server/src/main/java/com/linecorp/centraldogma/server/internal/replication/ZooKeeperCommandExecutor.java

Line 436 in 3e936b4

Gauge.builder("replica.has.leadership", this,

In order to troubleshoot this issue, I propose that we add a metric which indicates who curator thinks the actual leader is. This way, we can narrow down the scope of the issue.

leaderSelector.setId(String.valueOf(replicaId()));
Gauge.builder("leader", () -> {
    try {
        // this may be blocking
        return Integer.valueOf(leaderSelector.getLeader().getId());
    } catch (Exception e) {
        return -1;
    }
});

We can also potentially log leaderSelector.getParticipants() since this basically returns the order which will receive leadership (the first element is the leader).

@ikhoon also gave a good idea to add logs for more curator callbacks.

e.g.

centraldogma/server/src/main/java/com/linecorp/centraldogma/server/internal/replication/ZooKeeperCommandExecutor.java

Line 200 in 3e936b4

//ignore