[Models] Use in-place adds in Idefics2Vision

[lgeiger]

Purpose

This changes the Idefics2VisionTransformer model to use in-place adds where possible. This is relevant since vision encoders currently run outside of torch compile so things like this won't be automatically optimised away. See also #18922.

Related to: #23884

Test Plan

vllm serve openbmb/MiniCPM-V-4_5 --trust-remote-code

Test Result

In an internal benchmark with many image inputs this has a surprisingly big performance impact. I'm seeing a 5.5% - 6.2% increase in throughput with the openbmb/MiniCPM-V-4_5 model running on a L40s GPU.

@DarkLight1337 let me know if you'd like me to do a bit of search and replace and apply similar changes in the other model definitions as well which might also benefit text only models when run in eager mode.

[gemini-code-assist]

Code Review

This pull request introduces a performance optimization in Idefics2VisionTransformer by replacing out-of-place additions with in-place additions. The changes in Idefics2VisionEmbeddings and Idefics2EncoderLayer are correct and safe, as they operate on locally-scoped tensors, avoiding side effects. This modification should lead to reduced memory allocation and improved throughput as described in the pull request, which is a valuable improvement for inference performance. The changes are well-implemented. As you suggested in the description, applying this optimization to other models in the codebase would likely be beneficial as well.

[DarkLight1337]

Thanks for the optimization. Let's open a separate PR with a benchmark for each model

[lgeiger]

Rebased to re-trigger CI...