Export TF/Tensorboard/TF Summaries to prometheus

[jlewi]

In TF TF.summary is the primary ways people export metrics tracking the performance of their models during training. This data can be visualized using tensorboard (see here).

A lot of the key signals; e.g. accuracy metrics are just time series.

Should we export summaries in prometheus format so that they can be visualized and collected using tools in the prometheus tool chain?

Potential Use Cases

• For hyperparameter tuning(Katib) we'd like a generic way for the HP parameter infrastructure to get model metrics; prometheus could provide a standard interface that abstracts the details of metrics in a particular framework (PyTorch vs. TF).

• We could use this data to support features like "Run until converged"

Implementation

I think implementation would be pretty straightforward we would just need a Python server to read TF.Events files and export metrics to prometheus.

[ddysher]

I like the idea but haven't thought carefully about the details.

Essentially, metrics generated from ML applications are event driven (e.g. one batch is done, here is the accuracy), which is not a major targeting use care for prometheus. A concrete question is at what frequency should we scrape the metrics? I don't know if there's any other solutions, but it occurs to me that simply exposing metrics is not enough, we should at least leverage the push gateway if we want to go down this path.

/cc @gaocegege @codeflitting

[jlewi]

I think other folks have implemented this using session_hook.
So maybe we just need a reusable session hook library to export metrics.

[gaocegege]

@jlewi Hi, is there any reference implementation about the session_hook approach?

[jlewi]

I think weights and biases had an example of this.

[gaocegege]

Some helpful links:

• https://github.com/tensorflow/models/blob/master/official/utils/logs/metric_hook.py

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.