Question: Can Context FMHA be used to implement Transformer in a vision encoder for multimodal models?

[lmcl90]

I see that the multi-model models in the example all use TensorRT directly to deploy vision encoders, why not use TensorRT-LLM? Are there known issues or challenges associated with integrating Context FMHA into visual encoders?

[QiJune]

Yes, you can try to use TensorRT-LLM for the vision encoders. We have Bert example, DiT example, and community also contribute a SDXL model. I think it's not hard to develop a ViT model.

[lmcl90]

@QiJune Thanks for your replay. I will have a try.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

[github-actions]

This issue was closed because it has been stalled for 15 days with no activity.