Skip to content

Metrics should follow data-flow #31

@mariusae

Description

@mariusae

In the current implementation of metrics (#18), metrics are collected and aggregated from all tasks involved at the completion of a session.Run. This has a few drawbacks. First, all tasks may not be available at the end of session.Run (e.g., due to failed machines). Second, it ties stats collection to evaluation: we may in the future, persist task outputs to, e.g., s3, and clear task outputs from individual machines.

The solution to this is clear: we can flow Scopes along with task outputs; the executor can then aggregate them as the data flows.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions