docs: Add how-to guide for aligning LLM-as-Judge #2348
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:XL
- Modified .gitignore to include only the .cursor/plans directory, removing unnecessary entries. - Streamlined the git-pr.md file by replacing the lengthy PR description template with concise instructions for creating a PR using the gh CLI, enhancing usability for contributors.
- Introduced a new module in `examples/ragas_examples/judge_alignment` that includes functions for loading datasets, running evaluations, and calculating alignment metrics. - Added baseline and improved accuracy metrics for judge evaluations. - Enhanced documentation with a module docstring outlining the available functions and metrics.
- Introduced a new evaluation module in `examples/ragas_examples/judge_alignment` that implements functions for loading datasets, executing evaluations, and calculating alignment metrics between LLM judges and human judgments. - Defined baseline and improved accuracy metrics for judge evaluations. - Enhanced module documentation with a comprehensive docstring explaining the evaluation process and available functions.
- Add error analysis section with false positive/negative breakdown - Include improved v2 prompt with abbreviation guide and evaluation approach - Add ground truth quality guidance and tips on prompt optimization - Update evals.py with complete v2 prompt and main_v2() function - Fix model name from gpt-5-mini to gpt-4o-mini - Add collapsible output examples for baseline and v2 results
sanjeed5
force-pushed
the
docs/align-llm-judge
branch
from
6015f1e to
cc6bd52
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue Link / Problem Description
Adds documentation and examples for aligning LLM-as-Judge evaluators with human expert judgments. This addresses a common challenge where LLM judges may not align well with human evaluations, leading to unreliable automated assessments.
Changes Made
docs/howtos/applications/align-llm-as-judge.mdexamples/ragas_examples/judge_alignment/evals.py: Baseline and improved judge metrics with alignment measurement__init__.py: Module initialization with main entry pointsTesting
How to Test
uv run python -m ragas_examples.judge_alignmentmake serve-docsand navigate to the new guide