Skip to content

Fix ckpt convert bug #9521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 29, 2024
Merged

Fix ckpt convert bug #9521

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2f9956a
refine log
zhangbo9674 Nov 8, 2024
ee5f151
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
zhangbo9674 Nov 29, 2024
6221b91
refine
zhangbo9674 Nov 29, 2024
ab1297c
refine
zhangbo9674 Nov 29, 2024
016c70f
refine
zhangbo9674 Nov 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions paddlenlp/trainer/utils/ckpt_converter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ def gen_metadata_and_prepare_source_state_dict(self):
malloc_size = 0
for opt_state_name, opt_state_value in optimizer_state_dict.items():
malloc_size += opt_state_value.numel() * opt_state_value.element_size()
malloc_size = malloc_size.numpy() / 2**20
malloc_size = malloc_size / 2**20
logger.debug(f"{malloc_size} MB of GPU memory were allocated.")

# merge sharding
Expand Down Expand Up @@ -555,7 +555,7 @@ def load_state_dict_and_rename(self):
for k, v in state_dict.items():
memory_size += v.numel() * v.element_size()

memory_size = memory_size.numpy() / 2**20
memory_size = memory_size / 2**20
logger.debug(
f"The current rank has finished loading the checkpoint file and has allocated {memory_size} MB of GPU memory."
)
Expand Down
1 change: 0 additions & 1 deletion scripts/distribute/ci_case_auto.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ function llama_case_list_auto() {
llama_dy2st_auto_bs4_bf16_DP1-MP1-PP4-SD2
llama_align_dygraph_dy2st_auto_bs2_bf16_DP2-MP1-PP1
llama_pir_auto_fuse_ffn_attention_qkv_MP2
llama_convert_hybrid_ckpt_to_auto_parallel_bs2_fp32_DP2-MP1-PP1
llama_align_dygraph_dy2st_pir_auto_bs2_bf16_DP2-MP2-PP1-SP
llama_align_dygraph_dy2st_pir_auto_bs2_bf16_DP2-MP2-PP2-SP
llama_align_dygraph_dy2st_pir_auto_grad_merge_bs2_fp32_DP1-MP1-PP1
Expand Down
Loading