Skip to content

[ASR] Chang memory allocator strategy to fix gpu training hang #2478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 29, 2022
Merged

[ASR] Chang memory allocator strategy to fix gpu training hang #2478

merged 1 commit into from
Sep 29, 2022

Conversation

Zth9730
Copy link
Contributor

@Zth9730 Zth9730 commented Sep 28, 2022

PR types

Bug fixes

PR changes

Others

Describe

default memeory allocator strategy may case gpu training hang
no OOM raised when memory exhaused

fix #2459

@Zth9730 Zth9730 added this to the r1.3.0 milestone Sep 28, 2022
@Zth9730 Zth9730 self-assigned this Sep 28, 2022
@mergify mergify bot added the Example label Sep 28, 2022
@Zth9730 Zth9730 changed the title [s2t] Chang memory allocator strategy to fix gpu training hang [ASR] Chang memory allocator strategy to fix gpu training hang Sep 28, 2022
zh794390558
zh794390558 approved these changes Sep 29, 2022
View reviewed changes
Copy link
Collaborator

@zh794390558 zh794390558 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zh794390558 zh794390558 merged commit a657cc3 into PaddlePaddle:develop Sep 29, 2022
@Zth9730 Zth9730 mentioned this pull request Dec 12, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zh794390558 zh794390558 zh794390558 approved these changes

Assignees

@Zth9730 Zth9730

Labels
Example
Projects
None yet
Milestone
r1.3.0
Development

Successfully merging this pull request may close these issues.

[S2T] aishell/asr1 训练过程卡住不动
2 participants
@Zth9730 @zh794390558