Added Sycophancy Evaluation Metric in SDK, FE, Docs

[yashkumar2603]

Details

Resolves #2520
This PR adds the SycEval metric for evaluating sycophantic behavior in large language models. The metric tests whether models change their responses based on user pressure rather than maintaining independent reasoning by presenting rebuttals of varying rhetorical strength.
It is based on this paper https://arxiv.org/pdf/2502.08177
as linked in the issue.

Key Features:

• Multi-step evaluation process: Initial classification then Rebuttal generation then Response evaluation then Sycophancy detection
• Configurable rebuttal types: Simple, ethos, justification, and citation-based rebuttals
• Context modes: In-context and preemptive rebuttal presentation
• Separate rebuttal model: Uses dedicated model (defaults to llama3-8b) to avoid contamination
• Binary scoring: Returns 0.0 (no sycophancy) or 1.0 (sycophancy detected)
• Detailed metadata: Includes initial/rebuttal classifications and sycophancy type

Implementation:

• SycEval class with sync/async scoring methods
• Response classification and parsing
• Error handling and validation for all classification types
• can be imported using from opik.evaluation.metrics import SycEval in SDK easily, I tried to follow the coding style of the project, and other things mentioned in the contributing doc.

Issues

I faced one problem, I wasnt able to figure out a way to add the different results found out by the sycophancy analysis, such as sycophancy_type into the scores category in FrontEnd, as that would have required a STRING type in the LLM_SCHEMA_TYPE
So I instead made those available on the SDK, but not on the frontend. Please suggest something to tackle this problem. Guide me to make the necessary improvements in PR.

Documentation

• Added comprehensive docstrings with usage examples
• Updated evaluation metrics documentation
• Added configuration parameter explanations
• Included research context and score interpretation guidelines (a little when needed)

Working Video

2025-06-29_23-50-51.mp4

/claim #2520

Edit: added working video I forgot to add

[yashkumar2603]

Hello, @vincentkoc please review and suggest changes if any. Also kindly help me understand the frontend issue mentioned above.

• Thank you 😃

[vincentkoc]

Hello, @vincentkoc please review and suggest changes if any. Also kindly help me understand the frontend issue mentioned above.

• Thank you 😃

Thanks! @yashkumar2603 the team will review and circle back.

[yaricom]

Hi @yashkumar2603 ! Thank you for your work on this PR — it looks very promising. I’ve left a few review comments. Additionally, I’d like to ask you to add a unit test for your metric that uses mocked model calls but verifies the scoring logic in both synchronous and asynchronous modes.

Please take a look how other LLM judge metrics are tested.

[yashkumar2603]

Thanks for the review @yaricom !!
I am glad you liked the work. I will surely take a look at the unit tests, fix the reviews and update the PR.
Thank you again for your time !!

[yashkumar2603]

I have added the unit tests and also made necessary changes based on the reviews.
Kindly review 🙏🏾

[yaricom]

@aadereiko Could you please take a look at frontend changes if you have any comments or suggestions.

[aadereiko]

@yaricom @yashkumar2603
The FE part looks good :)

[yashkumar2603]

I have made the changes mentioned in the above comment, moved the test from integration to unit.
You are right, I had misplaced it. Thank you for pointing out.
Kindly review, merge it.

Thank you for your time.

[yaricom]

Dear @yashkumar2603 ! Thank you for committing the changes. Please run all tests locally using the OPIK server to ensure there are no unexpected errors. You can find detailed instructions on how to run the OPIK server here: https://www.comet.com/docs/opik/quickstart

[andrescrz]

Hi @yashkumar2603

Thank you for your contribution! 🙏 It looks like there are some merge conflicts that need to be resolved before we can continue. When you have a chance, could you please update the branch? Once the conflicts are resolved, we’ll be happy to provide a new review.

Let us know if you have any questions or need assistance!

[yashkumar2603]

Thank you for your reviews @yaricom @andrescrz 😃
I will surely make the changes, resolve conflicts and then update the PR. Been a busy for a few days, I will surely do this when i get the time.

[yashkumar2603]

Hello all!!
I have updated the code based on recommendations by @yaricom and also resolved the merge conflicts from main.
The llama-3 error was coming in tests because i was teesting with it on my local and forgot to change it to the model mentioned in the original paper. Really sorry for the confusion.

Please review @andrescrz.

Thank you for your time 🙏🏾

[yashkumar2603]

Sorry for the delays, I have fixed the examples.
Also added the comment explaining how to choose the model for rebuttal if one wishes to change it.
The idea is that,
rebuttal_model should be a widely available model from a different family than the model being tested to prevent evaluation leakage, as was the methodology in the paper. Using a model like gpt-3.5-turbo is a suitable and compatible choice.

The working tree got messed up a bit, as it was my first experience with merge conflicts. Really sorry for that, still learning.
Thank you for your time!! 🙏🏾

[yaricom]

Hi @yashkumar2603 ! Thank you for the commit.

Please fix the following frontend linter errors:

Run npm run lint

> [email protected] lint
> eslint src --max-warnings=0 && stylelint src/**/*.{css,scss,sass}


/home/runner/work/opik/opik/apps/opik-frontend/src/constants/llm.ts
Error:   374:17  error  Insert `⏎·····`      prettier/prettier
Error:   415:21  error  Insert `⏎·········`  prettier/prettier

✖ 2 problems (2 errors, 0 warnings)
  2 errors and 0 warnings potentially fixable with the `--fix` option.

Error: Process completed with exit code 1.

https://github.com/comet-ml/opik/actions/runs/16720751857/job/47324614565?pr=2624

Please fix the SDK linter errors:

https://github.com/comet-ml/opik/actions/runs/16720751851/job/47324614555?pr=2624
You need to install pre-commit as described in Contribution guide and run it locally:

cd sdks/python
pre-commit run --all-files