Skip to content

【Hackathon 5th No.64】PaddleNLP套件模型接入动转静训练功能 -part #7576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 18, 2023
Merged

【Hackathon 5th No.64】PaddleNLP套件模型接入动转静训练功能 -part #7576

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1a549e7
fix
MayYouBeProsperous Dec 4, 2023
e02cca9
add to_static for Trainer
MayYouBeProsperous Dec 13, 2023
775c0b2
fix
MayYouBeProsperous Dec 13, 2023
4a27ea5
codestyle
MayYouBeProsperous Dec 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions model_zoo/bert/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@ python -m paddle.distributed.launch --gpus "0" run_pretrain.py \
--weight_decay 1e-2 \
--adam_epsilon 1e-6 \
--warmup_steps 10000 \
--num_train_epochs 3 \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

num_train_epochs 参数脚本中未用到，设了会报错，所以删了

--input_dir data/ \
--output_dir pretrained_models/ \
--logging_steps 1 \
Expand All @@ -83,7 +82,6 @@ python -m paddle.distributed.launch --gpus "0" run_pretrain.py \
- `weight_decay` 表示AdamW优化器中使用的weight_decay的系数。
- `adam_epsilon` 表示AdamW优化器中使用的epsilon值。
- `warmup_steps` 表示动态学习率热启的step数。
- `num_train_epochs` 表示训练轮数。
- `input_dir` 表示输入数据的目录，该目录下所有文件名中包含training的文件将被作为训练数据。
- `output_dir` 表示模型的保存目录。
- `logging_steps` 表示日志打印间隔。
Expand Down Expand Up @@ -128,7 +126,6 @@ python -m paddle.distributed.launch --xpus "0" run_pretrain.py \
--weight_decay 1e-2 \
--adam_epsilon 1e-6 \
--warmup_steps 10000 \
--num_train_epochs 3 \
--input_dir data/ \
--output_dir pretrained_models/ \
--logging_steps 1 \
Expand All @@ -146,7 +143,6 @@ python -m paddle.distributed.launch --xpus "0" run_pretrain.py \
- `weight_decay` 表示AdamW优化器中使用的weight_decay的系数。
- `adam_epsilon` 表示AdamW优化器中使用的epsilon值。
- `warmup_steps` 表示动态学习率热启的step数。
- `num_train_epochs` 表示训练轮数。
- `input_dir` 表示输入数据的目录，该目录下所有文件名中包含training的文件将被作为训练数据。
- `output_dir` 表示模型的保存目录。
- `logging_steps` 表示日志打印间隔。
Expand Down
4 changes: 4 additions & 0 deletions paddlenlp/trainer/trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,10 @@ def __init__(
if model is None:
raise RuntimeError("`Trainer` requires either a `model` or `model_init` argument")

if self.args.to_static:
model = paddle.jit.to_static(model)
logger.info("Successfully to apply @to_static to the whole model.")

if self.args.should_save or self.args.should_save_model_state:
os.makedirs(self.args.output_dir, exist_ok=True)

Expand Down
4 changes: 4 additions & 0 deletions paddlenlp/trainer/training_args.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -705,6 +705,10 @@ class TrainingArguments:
default=False,
metadata={"help": "Whether to unify hybrid parallel checkpoint."},
)
to_static: Optional[bool] = field(
default=False,
metadata={"help": "Enable training under @to_static."},
)

def __post_init__(self):
env_local_rank = int(os.environ.get("PADDLE_RANK_IN_NODE", -1))
Expand Down