Update tokenization_bertweet.py (huggingface#16941)

datquocnguyen · elusenji · commit 06667a2869f1 · 2022-06-12T11:14:27.000+02:00
The emoji version must be either 0.5.4 or 0.6.0. Newer emoji versions have been updated to newer versions of the Emoji Charts, thus not consistent with the one used for pre-processing the pre-training Tweet corpus (i.e. not consistent with the vocab).
diff --git a/src/transformers/models/bertweet/tokenization_bertweet.py b/src/transformers/models/bertweet/tokenization_bertweet.py
@@ -152,7 +152,7 @@ def __init__(
             self.demojizer = demojize
         except ImportError:
             logger.warning(
-                "emoji is not installed, thus not converting emoticons or emojis into text. Please install emoji: pip3 install emoji"
+                "emoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0"
             )
             self.demojizer = None
 

Original file line number	Diff line number	Diff line change
`@@ -152,7 +152,7 @@ def __init__(`
`152`	`152`	`self.demojizer = demojize`
`153`	`153`	`except ImportError:`
`154`	`154`	`logger.warning(`
`155`		`- "emoji is not installed, thus not converting emoticons or emojis into text. Please install emoji: pip3 install emoji"`
	`155`	`+ "emoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0"`
`156`	`156`	`)`
`157`	`157`	`self.demojizer = None`
`158`	`158`