Add Gemini judge support with weighted scoring in G-Eval

[bofenghuang]

Hello 👋,

This PR enables the use of Gemini models as judge to get weighted summed scores in G-Eval.

Currently, there isn't a generate_raw_response like function to get logprobs for Gemini, so it falls back to generate function, which only produces the final sampled score. This PR:

• Adds generate_raw_response and a_generate_raw_response function
• Caps top_logprobs at 19, since Gemini only supports a range of [0, 20)
• Adds a transform_gemini_to_openai_like function to convert Gemini output to OpenAI format, so we can reuse the existing post-processing code (calculate_weighted_summed_score)
• Adds an initial unit test

One issue is that Gemini tokenizes the default rubric upper bound 10 to 0 and 1, so with current version of calculate_weighted_summed_score, we need to set the upper bound to less than 10

[vercel]

@bofenghuang is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

[penguine-ip]

Hey @bofenghuang thanks for the PR! Can we add a quick test for G-Eval as well (like supply the gemini model in code to G-Eval?) Thanks!

[bofenghuang]

Hey @penguine-ip ,

Thanks for your review!

Just rebased and added a mocked unit test plus a live test, where the live one reads the Gemini API config from .deepeval/.deepeval.

[bofenghuang]

Hi @penguine-ip , is this ready to be merged? Thanks!

[penguine-ip]

Hey @bofenghuang yes just one more thing. Would it be better to raise an attribute error if the function does fail? Since right now in G-Eval we are catching that gracefully. Let me know your thoughts

[bofenghuang]

Hey @penguine-ip,
IMO these fallbacks are very useful, e.g., falling back to a_generate if a_generate_raw_response doesn't exist, or to the greedy score if calculate_weighted_summed_score fails.
But I'd prefer to have a trace instead of silently falling back, since it took me a while as a new user to realize that Gemini didn't work with the weighted summed score. Maybe could add this to verbose_logs?

[penguine-ip]

Hey @bofenghuang yes actually that woul dbe craet. We can add a things like `"Score: X (weighted summation=True)", does this sound feasible?

[bofenghuang]

Yes, that would be great! Maybe we could do this in another PR.

[bofenghuang]

Hi @penguine-ip, do you think this PR is ready to merge? Anything you'd like to add? Thanks!

[bofenghuang]

Hey @penguine-ip, does this added warning look good to you?

[penguine-ip]

Hey @bofenghuang is it possible to remove the warning? It will scare users off. The Score: X (weighted=true/false) would be good enough

[bofenghuang]

Hey @penguine-ip, I reverted it. Could you add it later? I'm not fully sure I get what you mean, and it's a bit out of scope for this PR 🙂

[bofenghuang]

Hello @penguine-ip, just rebased the PR due to a conflict. Could you add the message? Thanks in advance.