-
Notifications
You must be signed in to change notification settings - Fork 1.1k
fix: enum values instead of names in testset synthesizers #2137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greptile Summary
This PR modifies the testset synthesizer components to use enum .value
instead of .name
when passing QueryLength
and QueryStyle
parameters to language model prompts. The change affects both single-hop and multi-hop query synthesizers in the ragas framework.
Specifically, the modification changes how enum parameters are passed to the prompt generation system:
- For
QueryLength
: Changes from passing uppercase names like 'LONG', 'MEDIUM', 'SHORT' to lowercase values like 'long', 'medium', 'short' - For
QueryStyle
: Changes from passing programmatic identifiers like 'MISSPELLED', 'PERFECT_GRAMMAR' to descriptive phrases like 'Misspelled queries', 'Perfect grammar'
This change integrates with the broader testset generation system where synthesizers create scenarios and samples for evaluating LLM applications. The BaseScenario
class contains QueryLength
and QueryStyle
enums that define query characteristics, and these synthesizers use these enums to generate appropriate test queries. By using the descriptive enum values, the system provides more meaningful context to the language models during query generation, which should improve the quality and accuracy of synthesized test data.
The change is applied consistently across both synthesizer types, indicating a deliberate architectural decision to standardize on using descriptive enum values throughout the query generation pipeline.
Confidence score: 4/5
- This change is relatively safe as it only modifies how enum values are passed to prompts without changing core logic
- The modification improves prompt clarity for language models, which should enhance generation quality
- Both affected files need attention to ensure the enum values are properly defined and consistent with the new usage pattern
2 files reviewed, no comments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix @lovets18 Could you please edit the commit msg to something more meaningful?
@anistark Thanks for the feedback! Updated the commit message to describe the change |
Thanks @lovets18 🎉 |
…radients#2137) **Change** Using Enum.value instead of Enum.name It seems that value (description) for QueryStyle is more logical than using names. The current name and value are almost identical, but this might change in the future Co-authored-by: valovtsov <[email protected]>
…radients#2137) **Change** Using Enum.value instead of Enum.name It seems that value (description) for QueryStyle is more logical than using names. The current name and value are almost identical, but this might change in the future Co-authored-by: valovtsov <[email protected]>
Change
Using Enum.value instead of Enum.name
It seems that value (description) for QueryStyle is more logical than using names. The current name and value are almost identical, but this might change in the future