[TRTLLM-5054][fix] Removing repeated loading of input processor

[rakib-hasan]

Initial implementation was preparing the multimodal inputs one at a time as it loads from the dataset. This caused the input processor to be re-initialized over and over for each input and takes a few minutes to load a 1000 entry dataset.

With the new approach, the whole list of multimodal inputs is passed to the fomatter once, to avoid repeated loading of the input processor.

[rakib-hasan]

/bot run

[tensorrt-cicd]

PR_Github #4596 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #4596 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #3310 completed with status: 'FAILURE'

[rakib-hasan]

/bot run

[tensorrt-cicd]

PR_Github #5404 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #5404 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3941 completed with status: 'SUCCESS'