Potentially lowercase the data in a lemmatizer if all of the training… #1331
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… data (or a user flag) requested it Testing additions: Add a basic unit test of the all_lowercase function Add a test of the caseless lemmatizer in the Pipeline Test that the Latin ITTB lemmatizer is marked as caseless. Check that the results for capitalized text is as expected Addresses #1330
AngledLuffa
force-pushed
the
dev
branch
2 times, most recently
from
cd3eeeb to
20a12c0
Compare
Jemoka
approved these changes
Jan 30, 2024
| """ | ||
| nlp = stanza.Pipeline('la', package='ittb', processors='tokenize,pos,lemma', model_dir=TEST_MODELS_DIR, download_method=None) | ||
| lemmatizer = nlp.processors['lemma'] | ||
| assert lemmatizer.config['caseless'] |
There was a problem hiding this comment.
Does the published models have this flag already, and it just poofed? Otherwise don't we need to release new models with this?
Lgtm otherwise
There was a problem hiding this comment.
I redid the Latin ITTB model with this flag. It's the only one so far, but all of the others should automatically pick it up if their entire dataset is lowercased. I do wonder if that will be surprising in the case of a Chinese model which accidentally gets used on some English text, though, for example
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Potentially lowercase the data in a lemmatizer if all of the training data (or a user flag) requested it
Testing additions:
Add a basic unit test of the all_lowercase function Add a test of the caseless lemmatizer in the Pipeline Test that the Latin ITTB lemmatizer is marked as caseless. Check that the results for capitalized text is as expected
Addresses #1330