[Feature Request] Count and truncate by tokens amount

[a-l-e-x-d-s-9]

Thank you for your great work!
I'm using EveryDream2trainer and it has a limit of 75 tokens per caption.
To check how many tokens are used I have to switch to:
https://github.com/AUTOMATIC1111/stable-diffusion-webui-tokenizer.git
Then go back and adjust the caption accordingly.
It would be great if the token counter will be presented near the captions, inside your extension.
Also if you could add an option to truncate the captions according to certain tokens number, it would be fantastic.
Thank you again for your amazing work!

[toshiaki1729]

I will try to implement the feature, after the refactoring of UI codes.
The UI-related codes are extremely tangled and the cleaning is in progress .

[a-l-e-x-d-s-9]

@toshiaki1729
Thank you!

[toshiaki1729]

I'm planning to add following features related to token count

• Show token count on the top-right of textboxes (like the prompt on text2img tab)
• Truncate by token count
• Token visualizer for selected image (like stable-diffusion-webui-tokenizer)
• Tag sort by token count + show tags with frequency, length, token count according to "Sort by" option (show nothing when alphabetical order is selected).

[a-l-e-x-d-s-9]

Cool.
Thank you!

[toshiaki1729]

Now, the feature to truncate tags by token amount is implemented in "Batch Edit Captions" > "Extras".