feat(models): introduce opt-in structured output interface and migrate initial metrics #2228
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add schema helpers to base models: generate_with_schema / a_generate_with_schema with graceful fallback when providers don’t accept `schema`. - Introduce capability flags via supports_structured_outputs / supports_json_mode for model opt-in. - OpenAI: opt-in based on allowlists - Tests: parameterized capability checks across all declared models via both sync/async schema-path tests for structured outputs and JSON mode asserting correct response_format and code path.
…sational GEval via model opt-in - Use generate_with_schema/a_generate_with_schema for steps and scoring - Normalize Pydantic vs JSON outputs consistently - Track evaluation cost where applicable - Remove legacy native/non-native branching and TypeError fallbacks
…tion helpers Refactored structured-output metrics to use the new model opt-in interface. - Unified cost calculation and structured-output fallback logic - Introduced gen_and_extract / a_gen_and_extract helpers for sync and async - Simplified schema extraction via consistent pattern across metrics The remaining metrics will be updated following the pattern established in this commit.
|
@BloggerBust is attempting to deploy a commit to the Confident AI Team on Vercel. A member of the Team first needs to authorize it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a unified opt-in interface for structured outputs across all model
providers.
Overview
generate_with_schema()anda_generate_with_schema()to the base modellayer with automatic fallback for providers that don’t yet support schema args.
supports_structured_outputs,supports_json_mode)for provider introspection and gradual rollout.
gen_and_extract()anda_gen_and_extract()helpers inmetrics/utils.py.In Progress ( this PR is in DRAFT )
Core benefit
This migration eliminates redundant parsing logic across metrics and models, establishes a unified structured output interface, and moves all provider specific behavior into the model layer. By introducing capability flags and opt-in schema helpers, metrics become provider agnostic, future proof, and free of branching logic. This design enables smarter routing, clearer telemetry, and ease of adoption of new provider capabilities without the need of further metric changes.