[Core] Increase default max_num_batched_tokens for multimodal models

[DarkLight1337]

Enabling chunked prefill causes some confusing errors for multimodal models as max_num_batched_tokens < num_multimodal_tokens leads to mismatched placeholder count when running the model.

This PR partially solves this issue by increasing the default max_num_batched_tokens for multimodal models so that it is sufficient for most cases.

As indicated by the TODO, it would be more ideal to determine the number of multimodal tokens that are in the prompt and raise an error if we detect that chunked prefill would truncate the multimodal tokens. However, this requires some refactoring for LLMEngine to access the multimodal registry used in the ModelRunner, so let's leave that to another PR.

As mentioned by @ywang96 , another improvement would be to dynamically set the default max_num_batched_tokens, but that also requires access to the ModelRunner as the maximum number of multimodal tokens is only available after init_mm_limits_per_prompt is called.

FIX #7996

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[ywang96]

LGTM! Thanks for the quick fix.

[DarkLight1337]

Hmm... seems that Fuyu test cannot run in CI with the increased max_num_batched_tokens=8192. (It works locally) I'll reduce it to 4096 then.