Skip to content
This repository was archived by the owner on Jan 15, 2024. It is now read-only.

Commit 277bd0d

Browse files
haven-jeoneric-haibin-lin
authored andcommitted
[Feature] Add SentencePieceTokenizer for BERT (#669)
* add BERTSPTokenizer * update doc * fix doc * add BERTVocab.from_sentencepiece() add test case * fix doctest * fix lint error * fix doctest and lint error * update for unicode processing(py2) * remove comments * disable lint error * fix flake8 error * make vocab consistent * remove six. make convert2unicode private
1 parent 8fe3f31 commit 277bd0d

File tree

5 files changed

+449
-113
lines changed

5 files changed

+449
-113
lines changed

0 commit comments

Comments
 (0)