Regarding the feasibility of multi-image data training and LoRA training

[fragrantly0202]

Great work, and thank you for open-sourcing it! I have a few questions:​​

​​First, since I currently need to use multiple images per sample, can the lmms-lab/M4-Instruct-Data format JSON file be directly used in scripts/train/finetune_ov.sh for training?​​
​​Second, does this work support LoRA training? My current understanding of finetune_ov.sh is that it only supports full-parameter fine-tuning. If there’s a way to apply LoRA for training the language model, I’d greatly appreciate your guidance!​​

Looking forward to your reply!

[Bhuvan-21]

You can directly use 'scripts/archived/finetune_lora.sh' with appropriate parameters to perform LoRA finetuning.

[HuangChiEn]

by the way, ov datasets are stored in parquet format, while llava-interleaved m4 (json file) doesn't. So it may also need to adjust the dataloader or change the dataset saving format.