Inquiry About History Retention In Training Process

[maojiaqi2324]

1. During the training process, do you retain the history of previous turns for the same scene? Is each turn treated independently during training?
2. If the history is retained, how is it implemented in the model? Are there any specific mechanisms or architectures that enable this?
3. Also, is there a plan to release the training code for this model? It would be helpful for further understanding and experimentation.

[rbler1234]

For the training phase on OST-Bench, we converted the data into an interleave format for convenience (the same format used for evaluating OST-Bench in VLMEvalKit). For each multi-turn dialogue within a scene (with n turns), we transform it into n samples of varying input lengths. Each sample contains the system prompt, all image inputs from the beginning up to the current turn, the current question, and the current answer (excluding questions and answers from previous turns).

To ensure consistency between the model’s training and evaluation input settings, we evaluate the trained models using the interleave version of OST-Bench provided in VLMEvalKit.

We have released the LLaVA-format training data (in interleave format) on HuggingFace. This format allows easy alignment with the input requirements of many mainstream models. We believe this may meet your needs.