Detect label inconsistency in SpanTask

[rmitsch]

Description

Check for inconsistency between specified labels and labels in examples.

Types of change

Checklist

• I confirm that I have the right to submit this contribution under the project's MIT license.
• I ran all tests in tests and usage_examples/tests, and all new and existing tests passed. This includes
  • all external tests (i. e. pytest ran with --external)
  • all tests requiring a GPU
• My changes don't require a change to the documentation, or if they do, I've added all required information.

[svlandeg]

We definitely need to think about how to handle such an inconsistency.

I think we should warn instead of error here though, and discard the examples that do not fall into the label list that was specified earlier. Basically that allows users to keep the list of examples fixed while playing with different labels for the LLM to recognize (sometimes adding/removing labels can influence accuracy within the same label category).

[svlandeg]

I think the refactor is fine, but did you also see my other comment?

I think we should warn instead of error here though, and discard the examples that do not fall into the label list that was specified earlier. Basically that allows users to keep the list of examples fixed while playing with different labels for the LLM to recognize (sometimes adding/removing labels can influence accuracy within the same label category).

Basically I think we can recover by just keeping to the pre-defined list of labels, and discard the non-matching ones, while warning?

[rmitsch]

Basically I think we can recover by just keeping to the pre-defined list of labels, and discard the non-matching ones, while warning?

Done in d2aadb3.

[svlandeg]

Virtually good to merge, just two smaller comments