implementation of broadcast div backward by reduce #37776

Zjq9409 · 2021-12-01T14:10:27Z

PR types

Performance optimization

PR changes

OPs

Describe

使用reduce实现broadcast div 反向，相比于原始性能数据如下：

case	pytorch(kernel)	优化前	优化前相比pytorch	优化后	优化后相比pytorch	加速比	优化前accuracy	优化后accuracy
[50, 128, 1000], [128, 1000]	0.46865	0.24259	优于 (48.24%)	0.33432	优于(28.66%)	0.73	2.56E+02	2.29E+05
[50, 128, 1000], [1, 128, 1000]	0.46940	0.24346	优于 (48.13%)	0.33333	优于(28.99%)	0.73	3.93E+05	8.19E+03
[16, 2048, 7, 7], [16, 2048]	0.14044	0.07819	优于 (44.32%)	0.10572	优于(24.72%)	0.74	1.28E+02	0.0
[16, 2048, 16, 16], [16, 2048, 16, 16]	0.71575	0.34497	优于 (1.07x)	0.34541	优于 (1.07x)	1.00	0.0	0.0
[16,1,513,513], [1]	0.31762	4.67214	差于 (13.71x)	0.22371	优于(29.57%)	20.88	2.44E-03	2.34E-02
[512, 896, 4, 12], [512, 896, 4, 1]	1.68353	2.82219	差于 (67.64%)	1.20446	优于(28.46%)	2.34	5.24E+05	0.0
[512, 896, 4, 12], [512, 896, 4, 1] fp16	1.17390	2.74304	差于 (1.34x)	0.78895	优于(32.79%)	3.48	0.0	0.0
[32, 12, 128, 128], [32, 1, 1, 128] fp16	0.34941	0.57034	差于 (63.23%)	0.20772	优于(40.55%)	2.75	0.0	0.0
[32, 1, 1, 128], [1, 12, 128, 1] fp16	0.38124	0.4983	差于 (30.71%)	0.25034	优于(34.34%)	1.99	0.0	0.0

其中case1、2、3相比于原始dev分支优化的比例有所下降，但是其他配置均有很大提升

paddle-bot-old · 2021-12-01T14:10:36Z

✅ This PR's description meets the template requirements!
Please wait for other CI results.

paddle-bot-old · 2021-12-01T14:10:38Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Zjq9409 · 2021-12-07T11:25:30Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

注释已删除

ZzSean · 2021-12-09T03:09:14Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

float和double代码重复

该函数目前已经删除

ZzSean · 2021-12-09T03:12:06Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

同上，float和double重复

ZzSean · 2021-12-09T06:09:08Z

paddle/fluid/operators/reduce_ops/reduce_functor_op.h

rebase一下最新代码

ZzSean · 2021-12-09T06:10:47Z

paddle/fluid/operators/elementwise/elementwise_functor.h

这个头文件有添加的必要吗

已经删除

ZzSean · 2021-12-09T06:12:25Z

paddle/fluid/operators/elementwise/elementwise_div_op.h

同sub，default_elementwise_div_grad和elementwise_div_grad cpu代码重复

ZzSean · 2021-12-09T06:14:06Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

t_y表示什么意思？变量名最好能直观体现出含义

ZzSean · 2021-12-09T06:15:58Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

函数命名遵循大驼峰规则，不要全大写

Done.该函数已经修改

ZzSean · 2021-12-09T06:19:27Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

注释位置调整一下，太不显眼了

ZzSean · 2021-12-09T06:21:03Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

这里求的是dx，函数名确实grady？不合适吧

该函数已经修改

ZzSean · 2021-12-09T06:22:03Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

变量名和注释位置同上

ZzSean · 2021-12-09T08:18:48Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

ZzSean · 2021-12-09T08:23:20Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

删除无用代码

* multithread_memory_optimize

* make some non_parallel unittest parallel execute * delete duplicate ut

…dlePaddle#37834)

* fix static git diff check * test=document_fix

* update logsumexp doc * update api doc * update api doc

* Debug * Fixed issue with reset_grad_inplace_version when used with clear_gradient & cross-batch accumulation * Rearranged interfaces * Fixed ci issues

…ja (PaddlePaddle#37890)

* add maxunpool2d in __all__ * fix MaxUnPool2D example

* add infrt code refined with Paddle's code style. * rename CinnRtConfig to InfRtConfig * rename CinnRt to InfRt of some code * rename CINNRT to INFRT * remove unnecessary code * replace CINN to INFRT in the source code * replace all "cinn" in code to "infrt" * remove some const_cast

…37824)

* fix CUDA Graph H2D bug again * fix no return bug

…addle#37897)

* Rearranged Eager AutoCodeGen directory structure * Removed USE_OP in Eager AutoCodeGen * Enabled generation for Operators without Grad/Inputs/Outputs * Resolved operators without input * Fixed merge conflicts * Enabled Eager AutoCodeGen for 10+ more operators

* refine a test case, test=develop * publish python c api for eager, test=develop * revert modify about test_allclose_layer.py, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * delete numpy includes, use pybind11 numpy.h, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * suport eager error msg, and add grad test case, test=develop * refine, test=develop * refine, test=develop * generate eager core ops, only 4 ops, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop * refine, test=develop

…int, this convert is wrong (PaddlePaddle#37929)

…addlePaddle#37931)

…37821) * Fix CUDAGraph bug for StreamSafeCUDAAllocator * Add CUDAGrapthAllocator check in multi-stream interface * Set FLAGS_use_stream_safe_cuda_allocator defaulted to false * Fix environment error for cmake * Fix cmake error * Add UT of GetAllocatorInterfaceTest * Add UT of CUDAGraphExceptionTest * Enhance CUDAGraphExceptionTest

* add boardcast_sub * add boardcast_sub

…le#37934)

* add update func of auto search * update unitest

ZzSean · 2021-12-10T07:09:18Z

paddle/fluid/operators/elementwise/elementwise_div_op.cu

inplace逻辑在上面的分支里也应该会存在吧，所以这里的代码能不能提取一下公共部分，两个分支的区别就是是否最后进行reduce

Zjq9409 force-pushed the broadcast_div_bw branch from b89d791 to 97e840c Compare December 3, 2021 08:08

Zjq9409 force-pushed the broadcast_div_bw branch from 97e840c to 1c656dc Compare December 3, 2021 09:21

Zjq9409 force-pushed the broadcast_div_bw branch from 1c656dc to 3087040 Compare December 3, 2021 09:43

Zjq9409 force-pushed the broadcast_div_bw branch from 3087040 to ab89c4d Compare December 5, 2021 09:13

Zjq9409 force-pushed the broadcast_div_bw branch from ab89c4d to a85db47 Compare December 6, 2021 06:15

Zjq9409 force-pushed the broadcast_div_bw branch from a85db47 to 2490a84 Compare December 6, 2021 07:35

Zjq9409 force-pushed the broadcast_div_bw branch from 2490a84 to 665694c Compare December 6, 2021 07:56

Zjq9409 changed the title ~~add broadcast_div_bw~~ implementation of broadcast div backward by reduce Dec 7, 2021

Zjq9409 commented Dec 7, 2021

View reviewed changes

paddle/fluid/operators/elementwise/elementwise_div_op.cu Outdated

Copy link

Contributor Author

Zjq9409 Dec 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

注释已删除

ZzSean reviewed Dec 9, 2021

View reviewed changes

JZZ-NOTE and others added 25 commits December 10, 2021 06:03

multithread memory optimize error fix (PaddlePaddle#37894)

4c4f6ef

* multithread_memory_optimize

make some non_parallel unittest parallel execute (PaddlePaddle#37805)

4896026

* make some non_parallel unittest parallel execute * delete duplicate ut

fix filter_by_instag op for lod_level=0 without lod;test=develop (Pad…

702b955

…dlePaddle#37834)

ut support block (PaddlePaddle#37909)

bdd4584

Fix static git diff (PaddlePaddle#37914)

c61080e

* fix static git diff check * test=document_fix

add cmake depend for api_gen.py (PaddlePaddle#37900)

0b1e9c6

update logsumexp doc (PaddlePaddle#37883)

930fb8e

* update logsumexp doc * update api doc * update api doc

Buf fix for reset grad inplace version (PaddlePaddle#37811)

d5d0bb7

* Debug * Fixed issue with reset_grad_inplace_version when used with clear_gradient & cross-batch accumulation * Rearranged interfaces * Fixed ci issues

block MASM : warning A4018 when building cryptopp in windows with nin…

f39bf5b

…ja (PaddlePaddle#37890)

add maxunpool2d in __all__ (PaddlePaddle#37698)

f81df79

* add maxunpool2d in __all__ * fix MaxUnPool2D example

[Fleet Executor] Add feed, fetch and check correctness (PaddlePaddle#…

3a30782

…37824)

bug fix for adamw (PaddlePaddle#37905)

abd3506

Fix CUDA Graph H2D bug by restore host memory (PaddlePaddle#37774)

dfd3c24

* fix CUDA Graph H2D bug again * fix no return bug

add pyyaml needed by python\paddle\utils\code_gen\api_gen.py (PaddleP…

d4c5adf

…addle#37897)

fix softmax max dim (PaddlePaddle#37901)

f9f1298

fix: when ceil_model==true && Padding_algo!=SAME, (x-size)/stride != …

d819760

…int, this convert is wrong (PaddlePaddle#37929)

add check whether tensor is inplace and leaf when calcute gradient (P…

c747edf

…addlePaddle#37931)

implementation of broadcast sub backward by reduce (PaddlePaddle#37754)

eafdf16

* add boardcast_sub * add boardcast_sub

[fleet_executor] bug fix for fleet_executor, test=allcase (PaddlePadd…

f52718b

…le#37934)

[fleet_executor] Add interceptor gc (PaddlePaddle#37889)

d0d5372

add update func of auto search (PaddlePaddle#37867)

d59ae10

* add update func of auto search * update unitest

ZzSean reviewed Dec 10, 2021

View reviewed changes

optimize code

d621924

Zjq9409 force-pushed the broadcast_div_bw branch from 665694c to d621924 Compare December 10, 2021 07:48

Zjq9409 closed this Dec 10, 2021

Zjq9409 mentioned this pull request Dec 10, 2021

implementation of broadcast div backward by reduce #38044

Merged

Uh oh!

implementation of broadcast div backward by reduce #37776

implementation of broadcast div backward by reduce #37776

Uh oh!

Conversation

Zjq9409 commented Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Describe

Uh oh!

paddle-bot-old bot commented Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paddle-bot-old bot commented Dec 1, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZzSean Dec 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZzSean Dec 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

80 participants

Zjq9409 commented Dec 1, 2021 •

edited

Loading

paddle-bot-old bot commented Dec 1, 2021 •

edited

Loading

ZzSean Dec 9, 2021 •

edited

Loading

ZzSean Dec 9, 2021 •

edited

Loading