question: does it work with PreTrainedTokenizerFast

[geraldstanje]

hi,

does it work with tokenizer class PreTrainedTokenizerFast and this tokenizer_config.json file? https://huggingface.co/answerdotai/ModernBERT-base/blob/main/tokenizer_config.json#L938

Thanks

[springkim]

@geraldstanje

Yes, FlashBertTokenizer optimizes tokenization by removing all unnecessary processes, and only provides training and inference functionality from Huggingface.

It can operate with just vocab.txt, and you only need to include do_lower_case, model_max_length, and tokenize_chinese_char in the constructor to match the tokenizer.config.

Furthermore, for speed improvement, rather than performing at the unicode level, it is optimized at the ANSI level, so the internal implementation itself can be considered different.

[geraldstanje]

@springkim thanks for quick reply. does it matter that this tokenizer config file has PreTrainedTokenizerFast in it? can i just read this file and run tokenizer without any issues?

[springkim]

@geraldstanje Yes, that's correct. Just like Hugging Face's tokenizer can perform encoding using a vocab.txt, BertTokenizerFlash can also support encoding in the same way.

from flash_tokenizer import BertTokenizerFlash
from transformers import BertTokenizer

titles = [
    '绝不能放弃，世界上没有失败，只有放弃。',
    'is there any doubt about it "None whatsoever"',
    "세상 어떤 짐승이 이를 드러내고 사냥을 해? 약한 짐승이나 몸을 부풀리지, 진짜 짐승은 누구보다 침착하지.",
    'そのように二番目に死を偽装して生き残るようになったイタドリがどうして初めて見る自分をこんなに気遣ってくれるのかと尋ねると「私が大切にする人たちがあなたを大切にするから」と答えては'
]

vocab_file = "vocab.txt"

tokenizer1 = BertTokenizerFlash(vocab_file, do_lower_case=False, model_max_length=512)
tokenizer2 = BertTokenizer(vocab_file, do_lower_case=False, model_max_length=512)

for title in titles:
    print(title)
    tokens1 = tokenizer1.tokenize(title)
    tokens2 = tokenizer2.tokenize(title)
    ids1 = tokenizer1(title, max_length=512, padding="longest").input_ids[0]
    ids2 = tokenizer2(title, max_length=512, padding="longest", return_tensors="np").input_ids[0].tolist()
    if tokens1 == tokens2 and ids1 == ids2:
        print("Accept!")
    else:
        print("Wrong Answer")
    print(tokens1)
    print(tokens2)
    print(ids1)
    print(ids2)
    print()

[geraldstanje]

@springkim in https://huggingface.co/answerdotai/ModernBERT-base/tree/main i dont see a vocab.txt - how does it work than?

i see these files only: tokenizer.json, tokenizer_config.json, special_tokens_map.json

[geraldstanje]

any idea @springkim about the above?

[springkim]

@geraldstanje

As you can see below, the tokenizer used by https://huggingface.co/answerdotai/ModernBERT-base is not the BertTokenizer.

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "answerdotai/ModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)

print(type(tokenizer))
# <class 'transformers.tokenization_utils_fast.PreTrainedTokenizerFast'>

[conceptofmind]

@geraldstanje

As you can see below, the tokenizer used by https://huggingface.co/answerdotai/ModernBERT-base is not the BertTokenizer.

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "answerdotai/ModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)

print(type(tokenizer))

<class 'transformers.tokenization_utils_fast.PreTrainedTokenizerFast'>

Hello,

Thank you for the great library.

Do you plan on supporting bpe tokenizers like the one mentioned above?

Thank you,

Enrico

[geraldstanje]

@springkim is there a way to convert the model answerdotai/ModernBERT-base tokenizer to a BertTokenizer?

[springkim]

@geraldstanje @conceptofmind

Yes, of course, we plan to support a variety of talkizers in the future. We're currently implementing BPE, so we'll look into it after that. Thanks for the great idea.

[conceptofmind]

@geraldstanje @conceptofmind

Yes, of course, we plan to support a variety of talkizers in the future. We're currently implementing BPE, so we'll look into it after that. Thanks for the great idea.

Will definitely be looking forward to that.

Thank you for all of your great work.