Skip to content

Support megatron dataset for T5 #6659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 13, 2023
Merged

Support megatron dataset for T5 #6659

merged 1 commit into from
Sep 13, 2023
+24 −15

Conversation

@LaiXinyi823
Copy link
Contributor

@LaiXinyi823 LaiXinyi823 commented Aug 9, 2023

PR types

New features

PR changes

APIs

Description

Support megatron dataset for T5

@paddle-bot
Copy link

paddle-bot bot commented Aug 9, 2023

Thanks for your contribution!

@codecov
Copy link

codecov bot commented Aug 9, 2023 •
edited
Loading

Codecov Report

Merging #6659 (6a41b11) into develop (e49842c) will decrease coverage by 0.17%.
Report is 2 commits behind head on develop.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop    #6659      +/-   ##
===========================================
- Coverage    60.06%   59.90%   -0.17%     
===========================================
  Files          552      554       +2     
  Lines        81755    81975     +220     
===========================================
  Hits         49105    49105              
- Misses       32650    32870     +220
Files Changed Coverage Δ
paddlenlp/experimental/transformers/__init__.py 0.00% <0.00%> (ø)
...dlenlp/experimental/transformers/bloom/__init__.py 0.00% <0.00%> (ø)
...dlenlp/experimental/transformers/bloom/modeling.py 0.00% <0.00%> (ø)
...erimental/transformers/fused_transformer_layers.py 0.00% <0.00%> (ø)
...enlp/experimental/transformers/generation_utils.py 0.00% <0.00%> (ø)

@LaiXinyi823 LaiXinyi823 force-pushed the develop branch 2 times, most recently from a47f2dc to 3bed5b1 Compare August 9, 2023 03:43
@CLAassistant
Copy link

CLAassistant commented Aug 9, 2023 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@LaiXinyi823 LaiXinyi823 force-pushed the develop branch 5 times, most recently from e843a53 to 37ce41e Compare August 9, 2023 09:12
ZHUI
ZHUI reviewed Aug 24, 2023
View reviewed changes
examples/language_model/t5/README.md Outdated
Copy link
Contributor

@ZHUI ZHUI Aug 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在数据ID化步骤中，我们需要配置tokenzer_name，选择t5模型对应的tokenizer；通过下面脚本转化，我们可以得到处理好的预训练数据，token ids:baike_sample_ids.npy, 文章索引信息baike_sample_idx.npz.（这里提供了一个处理好的预训练数据，可点击链接下载）

这块需要搞一个样例数据出来

KB-Ding
KB-Ding reviewed Sep 5, 2023
View reviewed changes
Copy link
Contributor

@KB-Ding KB-Ding left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. 目前有冲突需要解决一下。
  2. 是否对齐新旧版本的数据？比如同一份数据处理出的旧版npy和新版bin，设定seed，跑前几步看看两版拿到的数据是否一样。

examples/language_model/t5/README.md Outdated
Copy link
Contributor

@KB-Ding KB-Ding Sep 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

补充一下mmap和lazy的区别：“mmap”格式在读入数据时会建立内存映射，“lazy”格式在读入数据时直接从文件读取。

Copy link
Contributor Author

@LaiXinyi823 LaiXinyi823 Sep 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改为：指定输入文件数据制作类型，默认为mmap，可指定mmap或lazy。“mmap”格式在读入数据时会建立内存映射，“lazy”格式在读入数据时直接从文件读取。

examples/language_model/t5/t5_run_pretrain_trainer.py Outdated
Copy link
Contributor

@KB-Ding KB-Ding Sep 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

建议和llama一致：help="mmap/lazy format converted from preprocessed data."

Copy link
Contributor Author

@LaiXinyi823 LaiXinyi823 Sep 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改为："help": "mmap/lazy format converted from preprocessed data."

model_zoo/ernie-1.0/args.py Outdated
Copy link
Contributor

@KB-Ding KB-Ding Sep 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

最好和llama保持一致，“mmap/lazy format converted from preprocessed data”

Copy link
Contributor Author

@LaiXinyi823 LaiXinyi823 Sep 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改为：help="mmap/lazy format converted from preprocessed data."

examples/language_model/t5/README.md Outdated
Copy link
Contributor

@KB-Ding KB-Ding Sep 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在数据准备中，预置的token ids:baike_sample_ids.npy, 文章索引信息baike_sample_idx.npz样例应改为bin格式与idx格式，数据制作可以参考这里，注意参数配置，参考这里

@LaiXinyi823 LaiXinyi823 force-pushed the develop branch 7 times, most recently from e7ab798 to 9cd67e7 Compare September 11, 2023 09:58
@LaiXinyi823
Copy link
Contributor Author

LaiXinyi823 commented Sep 11, 2023

  1. 目前有冲突需要解决一下。
  2. 是否对齐新旧版本的数据？比如同一份数据处理出的旧版npy和新版bin，设定seed，跑前几步看看两版拿到的数据是否一样。
  1. 冲突已解决
  2. 测试跑几个step，新旧版本数据相同，loss一致。

gongel
gongel reviewed Sep 12, 2023
View reviewed changes
Copy link
Member

@gongel gongel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

paddlenlp/data/indexed_dataset.py没看到任何改动，可以不用放在 PR 里

examples/language_model/t5/README.md Outdated
Copy link
Member

@gongel gongel Sep 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里为啥出来的是 gpt prefix的数据集？

Copy link
Contributor Author

@LaiXinyi823 LaiXinyi823 Sep 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

最后t5用的是gpt的openwebtext数据集，所以prefix是gpt，改成t5吗？

Copy link
Member

@gongel gongel Sep 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

用户按你的步骤跑出来的数据集是什么名称呢？保持一致，要可复现。

Copy link
Contributor Author

@LaiXinyi823 LaiXinyi823 Sep 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

用户按你的步骤跑出来的数据集是什么名称呢？保持一致，要可复现。

已修改

@LaiXinyi823
Copy link
Contributor Author

LaiXinyi823 commented Sep 12, 2023

paddlenlp/data/indexed_dataset.py没看到任何改动，可以不用放在 PR 里

ok

@LaiXinyi823 LaiXinyi823 force-pushed the develop branch 6 times, most recently from 06d6b9a to 673d356 Compare September 12, 2023 15:49
@LaiXinyi823 LaiXinyi823 force-pushed the develop branch 2 times, most recently from 1c9e3c4 to d225406 Compare September 12, 2023 16:13
@LaiXinyi823
fix T5 readme
6a41b11 
Support megatron dataset for T5
@gongel gongel merged commit a43138b into PaddlePaddle:develop Sep 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ZHUI ZHUI ZHUI left review comments

@gongel gongel gongel approved these changes

+1 more reviewer

@KB-Ding KB-Ding KB-Ding approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@LaiXinyi823 @CLAassistant @ZHUI @gongel @KB-Ding