Fix custom tokenizers test #19052
Merged
Fix custom tokenizers test #19052
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sgugger
commented
Sep 15, 2022
| elif is_tf_available(): | ||
| returned_tensor = "tf" | ||
| else: | ||
| elif is_flax_available(): |
There was a problem hiding this comment.
This test was failing when no framework (PyTorch, TF, Flax) is installed. Returning instead of failing now.
|
The documentation is not available anymore as the PR was closed or merged. |
ydshieh
approved these changes
Sep 15, 2022
There was a problem hiding this comment.
LGTM, thanks.
But I don't understand well this test: why we test tokenization for these 3 models only?
Also with a quick search, I can see @custom_tokenizers decorator is applied to cpm and BertJapanese. But here we run tests for openai and clip?
I must miss context.
|
@ydshieh Those three files contain tests that require some specific dependencies to be installed (ftfy for openai and clip). So in the other test jobs, those tests are never run. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The custom tokenizers tests were never run because the test fetcher was not run before we look at its output to decide whether or not to run the tests. This PR fixes that and also adds missing tests to the nightly suite.