Skip to content

[Fleet Executor] Construct runtime graph #37158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 17, 2021
Merged

[Fleet Executor] Construct runtime graph #37158

merged 2 commits into from
Nov 17, 2021

Conversation

@LiYuRio
Copy link
Contributor

@LiYuRio LiYuRio commented Nov 12, 2021 •
edited by FeixLiu
Loading

PR types

New Features

PR changes

Others

Describe

创建运行时图

@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

FeixLiu
FeixLiu reviewed Nov 15, 2021
View reviewed changes
FeixLiu
FeixLiu reviewed Nov 15, 2021
View reviewed changes
@FeixLiu
Copy link
Contributor

FeixLiu commented Nov 15, 2021

在关键的地方加一写VLOG（3）的输出用来debug吧。比如推导依赖的部分，比如interceptor_id 与 task_id 、rank等映射的部分。

FeixLiu
FeixLiu reviewed Nov 16, 2021
View reviewed changes
wangxicoding
wangxicoding reviewed Nov 16, 2021
View reviewed changes
paddle/fluid/distributed/fleet_executor/fleet_executor_desc.proto Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

后面复用distributed_strategy是不是更好些，可能还会有sharding_degree

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

因为distributed_strategy.proto在framework目录下，和这个proto不在一个文件夹，在当前文件夹下的CMakeList里调用generic.cmake里定义的proto_library函数，会将protobuf的搜索路径设置为当前文件夹，同时protobuf的import不支持相对路径，所以暂时没想到怎么直接引用distributed_strategy.proto里的定义。

paddle/fluid/distributed/fleet_executor/runtime_graph.cc Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dp、pp、mp以后的顺序可能会变

paddle/fluid/distributed/fleet_executor/runtime_graph.cc Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

建议把dp_degree、pp_degree、mp_degree这几个封装为一个结构体，当做笛卡尔坐标系，然后加上进程rank号和笛卡尔坐标系的相互转换，可能简洁一点点。然后顺序的问题加个映射也很容易解决
{x, y, z} = rank2coord(pid);
left_x = (x - 1 + xranks) % xranks; left_rank = coord2rank({left_x, y, z})

@FeixLiu FeixLiu requested a review from wangxicoding November 16, 2021 10:53
wangxicoding
wangxicoding approved these changes Nov 16, 2021
View reviewed changes
Copy link
Contributor

@wangxicoding wangxicoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@FeixLiu FeixLiu merged commit 0daa69d into PaddlePaddle:develop Nov 17, 2021
@LiYuRio LiYuRio deleted the runtime_graph branch November 17, 2021 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FeixLiu FeixLiu FeixLiu left review comments

@wangxicoding wangxicoding wangxicoding approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@LiYuRio @FeixLiu @wangxicoding