Which collator to use for padding-free packing: WithFlattening or Default LanguageModelling?

[jiosephlee]

Hi!

To my knowledge, it seems like there are two DataCollators available that can handle, one, packed examples and, two, padding-free attention.

WithFlattening: https://github.com/huggingface/transformers/blob/67ddc82fbc7e52c6f42a395b4a6d278c55b77a39/src/transformers/data/data_collator.py#L1973

custom LanguageModelling in sft_trainer.py:

trl/trl/trainer/sft_trainer.py

Line 109 in 686cd35

class DataCollatorForLanguageModeling(DataCollatorMixin):

Based on documentation it seems like WithFlattening has a nice feature of preventing the last token of a packed example from predicting the first token of the next example. Otherwise, not sure what the differences from an initial reading of the code.

[LeonEricsson]

Padding-free training is not supported when passing a custom collator, such as WithFlattening. You have to use DataCollatorForLanguageModelling; which, like WithFlattening, avoids cross-contamination between samples as of #3526

[jiosephlee]

Hi Leon!

Thank you for answering. I'm curious if @qgallouedec @thepowerfuldeez has any insights on this, but it seems that the default choice for the data collator used to be WithFlattening when padding_free is True 30fed95 (since it was meant to address the issue of padding-free training per huggingface/transformers#31629) . I'm curious about the reason behind this change.