This repository was archived by the owner on Sep 10, 2025. It is now read-only.

Tokenizers cpp 1251#1379

Closed

gabe-l-hart wants to merge 35 commits intopytorch:mainfrom

gabe-l-hart:TokenizersCpp-1251

Commits on Nov 15, 2024

feat(models): Add models.json blocks for Granite Code 3b and 8b
gabe-l-hart
committed
feat: Initial model params for granite code 3b
gabe-l-hart
committed
fix(model config): Fix model configs for Granite Code
gabe-l-hart
committed
feat(granite): Add model params for granite-code-8b
gabe-l-hart
committed
fix(deps): Add tokenizers to the deps explicitly
gabe-l-hart
committed
feat(tokenizer): Add basic support for jinja2 template rendering for HF tokenizers
gabe-l-hart
committed
fix(chat): Add HFTokenizerChatFormatter and use it for HF tokenizers
gabe-l-hart
committed
fix(deps): Add jinja2 as an explicit dep
gabe-l-hart
committed
refactor(tokenizer): Add const qualifiers to const methods in tokenizer classes
gabe-l-hart
committed
feat(tokenizers): Add submodule for nlohmann/json
gabe-l-hart
committed
feat(tokenizer): Add stub of TokenizersTokenizer
gabe-l-hart
committed
feat(tokenize_tool): Add a simple tool to round-trip tokenize some strings
gabe-l-hart
committed
feat(tokenizers): Mostly complete impl of loading tokenizer.json/tokenizer_config.json
gabe-l-hart
committed
fix(tokenize_tool): Fix logic for decoding if encoded is empty
gabe-l-hart
committed
feat(tokenize)!: Add gtest submodule
gabe-l-hart
committed
feat(tokenize): Initial work for pre-tokenizer class hierarchy with tests
gabe-l-hart
committed
chore(gitignore): Ignore the build dir (standard CMake build dir name)
gabe-l-hart
committed
test: Extract testing into a helper function
gabe-l-hart
committed
WIP!: Initial work towards making the tiktoken implementation a generic BPE
gabe-l-hart
committed
feat(tokenizer): Add Digit pre tokenizer
gabe-l-hart
committed
feat(tokenizers): Add a vendored copy of the unicode support from llama.cpp
gabe-l-hart
committed
fix(tokenizers): Change the return type of pre_tokenize to allow ownership
gabe-l-hart
committed
feat(tokenizers): Add byte-level pre-tokenizer
gabe-l-hart
committed
fix(tokenizers): Use pass-by-value for string views
gabe-l-hart
committed
feat(tokenizers): add SequencePreTokenizer
gabe-l-hart
committed
feat(tokenizers): Add the PreTokenizerConfig factory class
gabe-l-hart
committed
fix(tokenizers): Fix naming for inheriting from Tiktoken
gabe-l-hart
committed
fix(tokenizer): TokenizersTokenizer -> HFTokenizer
gabe-l-hart
committed
fix(tokenizer): Remove unnecessary templates on Tiktoken impl methods
gabe-l-hart
committed
feat(tokenizer): Split Tiktoken out into BPETokenizerBase and Tiktoken
gabe-l-hart
committed
feat(tokenizers): Add json parsing to PreTokenizerConfig factory
gabe-l-hart
committed
feat(tokenizers): Use BPETokenizerBase and PreTokenizers in HFTokenizer
gabe-l-hart
committed
feat(tokenizers): Add a minimal implementation of the HF decoder suite
gabe-l-hart
committed
fix(comment): Better comment in pre_tokenizer.h
gabe-l-hart
committed
feat(tokenizers): Add decoding support to HFTokenizer
gabe-l-hart
committed

Tokenizers cpp 1251#1379gabe-l-hart wants to merge 35 commits intopytorch:mainpytorch/torchchat:mainfrom gabe-l-hart:TokenizersCpp-1251gabe-l-hart/torchchat:TokenizersCpp-1251Copy head branch name to clipboard

Commits

Commits on Nov 15, 2024

Tokenizers cpp 1251#1379
gabe-l-hart wants to merge 35 commits intopytorch:mainfrom
gabe-l-hart:TokenizersCpp-1251