[AI Evaluation] EquivalenceEvaluator is not producing an answer

[mikeholczer]

Description

When attempting to use the EquivalenceEvaluator, my testing comes back Inclusive. I did some debugging and I found that if using the debugger increase the MaxOutputTokens from 1 to 2, it starts working as I'd expect. I'm not sure why it needs more than 1 output token since the response should be a number between from 1-5, but that's what I'm seeing.

"Failed to parse numeric score for 'Equivalence' from the following text:"

https://github.com/dotnet/extensions/blob/f04bd506a04b585486be20fb7b08157162f2f527/src/Libraries/Microsoft.Extensions.AI.Evaluation.Quality/EquivalenceEvaluator.cs#L55C14-L55C33

Reproduction Steps

var messages = new List<ChatMessage>
{
    new(ChatRole.System, "Your are a helpful assistant."),
    new(ChatRole.Assistant, "What's the 3rd planet from the sun?")
};

var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, "The Earth is the 3rd planet."));

var chatConfig = new ChatConfiguration(chatClient.AsBuilder().Build());

var equivalenceEvaluatorContext = new EquivalenceEvaluatorContext("The 3rd planet from the sun is the Earth.");

var equivalenceEvaluator = new EquivalenceEvaluator();
var evaluationResult =
    await equivalenceEvaluator.EvaluateAsync(messages, response, chatConfig, additionalContext: [equivalenceEvaluatorContext]);

Debug.WriteLine(evaluationResult.Metrics.Single().Value.Diagnostics?.Single().Message);

Expected behavior

evaluationResult.Metrics.Single().Value.Interpretation.Rating to be Exceptional

Actual behavior

evaluationResult.Metrics.Single().Value.Diagnostics?.Single().Message is: Failed to parse numeric score for 'Equivalence' from the following text: When looking in the debugger one can see that it model stopped generating output because of the max token limit being hit.

Regression?

No response

Known Workarounds

Attached the debugger here, and change the _clientOptions.MaxOutputTokens to 2 before it's used.

Configuration

Using .net 9, and getting the IChatClient from AnthropicClient.Messages.AsBuidler().Build() with the Anthropic.SDK nuget package and using the AnthropicModels.Claude4Sonnet model id.

Other information

No response

[mikeholczer]

I switched over to use the gpt-4o model through GitHub Models, since that's the model that the comments say the evaluator was testing with and it works correctly with it, so this may be a difference with the Anthropic model's output token counting or usage, but still since the evaluator is intended to be model agnostic, still seems like an issue.

[shyamnamboodiripad]

Thanks for reporting and figuring out the root cause @mikeholczer. This is interesting - I wonder why Claude requires two tokens to represent a single digit.

EquivalenceEvaluator is currently probably the only remaining evaluator in the Quality package that sets MaxOutputTokens to 1. Some of the other evaluators used to do the same - however, their prompts were subsequently updated to produce larger responses (that now also include LLM reasoning).

The prompts that these evaluators use are kept in sync with python counterparts in the Azure SDK for Python, and the python counterpart for this evaluator also sets max_tokens to 1. However, the python evaluators above are generally not intended to be used with models not hosted on Azure.

Increasing the MaxOutputTokens to a slightly larger number seems like it may be a good idea in this case. FYI @peterwald

[mikeholczer]

@shyamnamboodiripad thanks for that context. Should I create a issue in the Azure/azure-sdk-for-python repo? Or is it OK to have the discussion here? Looks like the related change there would be in: https://github.com/Azure/azure-sdk-for-python/blob/92e25aca40a35a2fb5d09dff62d28a276010f348/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/similarity.prompty#L4-L13

[shyamnamboodiripad]

@mikeholczer Nope - this is still the right place to discuss the .NET evaluators in the Microsoft.Extensions.AI.Evaluation libraries. I was merely pointing to the linked Azure repo for context on why things are the way they are :) As I mentioned above, the Azure SDK is mainly intended to work with (OpenAI) models hosted on Azure, so we shouldn't need changes there.

While the prompts for the .NET evaluators in the current repo are kept in sync with those in the Azure repo, I think we can be a little more flexible with aspects like MaxOutputToken. I have created #6835 to bump up the limit.