[BFCL] score discrepancy for gpt-4o-2024-11-20-FC and gpt-4o-mini-2024-07-18-FC

[Skripkon]

I run the evaluation for several times with zero change in this awesome repository.

My results on LIVE_SIMPLE for gpt-4o-mini-2024-07-18-FC: 79.457 %

Leaderboard says: 69.77 %

The same issue with the gpt-4o-2024-11-20-FC (about +10% accuracy in my runs)

[HuanzhiMao]

Hi @Skripkon,

Could you pull the latest main and try again? On the current main (d85e9ad), I re-ran gpt-4o-mini-2024-07-18-FC on LIVE_SIMPLE and got 69.38%, which is in line with the leaderboard score. A ~10pp lift usually points to a config mismatch.

[Skripkon]

@HuanzhiMao
Hi, I cloned the repo once again to ensure I'm running the last version.

The only changes I made:

1. set my API key + base url
2. switch from OpenAIResponsesHandler to OpenAICompletionsHandler

And I still have 🎯 Accuracy: 79.46% for gpt-4o-mini-2024-07-18-FC.

It must be due to the usage of OpenAICompletionsHandler .
Is this expected?

[Skripkon]

OKAY.
I tried to use OpenAIResponsesHandler and it is in line with the leaderboard (🎯 Accuracy: 70.16% for live_simple).

There must be an explanation for it.

[Skripkon]

GPT-4o-mini-2024-07-18-FC, Irrelevance Detection, OpenAIResponsesHandler:

🎯 Accuracy: 92.56

Leaderboard: 80.37