[Misc] Refactor tokenizer interface

[DarkLight1337]

Purpose

• Consolidate AnyTokenizer and TokenizerBase by making TokenizerBase a Protocol, and rename both of them to TokenizerLike (with back-compatibility)
• Remove unused attributes from TokenizerBase: sep_token, pad_token, encode_one
• Move MistralTokenizer, TokenizerLike, TokenizerRegistry into vllm.tokenizers
• Fix various type checking issues associated with replacing AnyTokenizer with TokenizerLike
• Move tests from tests/tokenization to tests/tokenizers and update Buildkite pipeline accordingly
• Consolidate related tests to fit the directory structure

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--29693.org.readthedocs.build/en/29693/

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a significant and well-executed refactoring that modernizes the tokenizer abstraction by introducing the TokenizerLike protocol. This change improves modularity, type safety, and flexibility across the codebase. The consolidation of tokenizer tests into a dedicated tests/tokenizers directory is also a great organizational improvement.

I've identified a couple of critical bug fixes related to handling cases where the tokenizer is not initialized (skip_tokenizer_init=True), which significantly improves the robustness of the OpenAI-compatible endpoints. These changes prevent potential crashes and provide clearer error messages to the user.

Overall, this is an excellent contribution that enhances the maintainability and reliability of vLLM's tokenization system. The changes are thorough and consistent.

[DarkLight1337]

Failing tests are known failures on main