Skip to content

Conversation

@sahusiddharth
Copy link
Contributor

@sahusiddharth sahusiddharth commented Oct 17, 2025

resolves: #5147

  • The criteria_evaluator_factory function returns type[ExperimentEvaluator], but MyPy cannot properly infer that the dynamically created class is actually instantiable (it sees it as abstract rather than concrete)
  • Used # type: ignore comments specifically for these factory-created evaluator instances to bypass MyPy limitations with dynamic class creation
  • Leveraged the create_evaluator decorator's type information to ensure proper typing for evaluator functions throughout the test suite

Note

Enables MyPy on experiments tests and updates evaluator tests to use the decorator APIs, adding targeted type ignores for dynamically created evaluators.

  • Tests / MyPy:
    • Remove tests/unit/datasets/test_experiments.py from tests/mypy.ini exclude, enabling type checks.
  • Experiments tests (tests/unit/datasets/test_experiments.py):
    • LLM evaluator instantiations: add # type: ignore to ConcisenessEvaluator(...) and HelpfulnessEvaluator(...) where dynamic types confuse MyPy.
    • Evaluator decorator tests:
      • Use create_evaluator(kind="CODE") and dict input signatures; assert via .evaluate(...).label.
      • Async variant uses .async_evaluate(...) with label assertions and preserved name/kind metadata.

Written by Cursor Bugbot for commit 19fa176. This will update automatically on new commits. Configure here.

@github-project-automation github-project-automation bot moved this to 📘 Todo in phoenix Oct 17, 2025
@dosubot dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Oct 17, 2025
@sahusiddharth
Copy link
Contributor Author

Hi @axiomofjoy,

When you get a moment, could you please take a quick look at this PR and let me know if there’s anything I should modify? Appreciate your time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files.

Projects

Status: 📘 Todo

Development

Successfully merging this pull request may close these issues.

[ci] add type checks to tests/unit/datasets/test_experiments.py

1 participant