Add support for pre-tokenized streaming dataset finetuning#601

boomanaiden154 wants to merge 11 commits intomosaicml:mainmosaicml/llm-foundry:mainfrom

boomanaiden154:ft-tokenized-prboomanaiden154/llm-foundry:ft-tokenized-prCopy head branch name to clipboard