[FR]: Online Evaluation custom code metric should support reading the whole trace input/output data rather than the specific string values

[Colstuwjx]

Proposal summary

As we can see, currently the online evaluation rule metric has ONLY supported to specific one input and one output string which referenced from the whole trace data. I would expect to read the whole trace data from the custom metric function so that it would be possible to analysis the trace data and generate a metric value for this time trace.

Motivation

While we didn't support this feature, it seems to be hard to write code to analysis the array result of output (for example) within the rule metric function.

[Colstuwjx]

After dive into the code, I've found the data which would send to python executor as param was defined as Map<String, String>, from my mind, there are two options for supporting this feature:

1. change the data to Map<String, Object> type so that it would store advanced data type, e.g. an array object. However, the end user may need to convert the input/output type to a detail type ?
2. pass the whole trace as an additional context to the custom metric code function so that it's possible to do some advanced process, e.g. extract the array output from the trace and evaluate the array output

Thanks.
cc @alexkuzmik @jverre @Lothiraldan

[aadereiko]

@Colstuwjx
Until it is implemented, you can use JSON serialization to access complete trace sections in your custom metrics and then use a custom metric

import json
from opik.evaluation.metrics import BaseMetric, ScoreResult

class FullTraceAnalysisMetric(BaseMetric):
    def __init__(self):
        super().__init__("full_trace_analysis")
    
    def score(self, full_input, full_output, full_metadata, **kwargs):
        input_data = json.loads(full_input) if full_input else {}
        output_data = json.loads(full_output) if full_output else {}
        metadata = json.loads(full_metadata) if full_metadata else {}
        
        if isinstance(output_data.get('results'), list):
            array_results = output_data['results']
            score = self.analyze_array_results(array_results)
        
        model_info = metadata.get('model_details', {})
        
        return ScoreResult(
            value=score,
            name=self.name,
            reason="Analyzed full trace structure"
        )
    
    def analyze_array_results(self, results):
        return len(results) / 10.0