Comparison of different solutions for deterministic backward

[defei-coder]

Thanks to @tridao for reviewing PR722 and mentioned a new solution.
In approach, @tridao proposed a new idea which calculate different local dQ partitioned based on the number of SMs and reduce sum them in later kernel. It is a concise solution, but the number of blocks is small, unable to fully utilize GPU resources results in poor performance.
I conducted a performance comparison between PR722 and approach. Based on the attention module end-to-end timing metric, PR722 is proved to be more efficient.
I could uderstand your design philosophy and agree that is a good solution, but in real case we found that PR722 is more efficient. We may need to testing with more cases to get the final conclusion. Thus I suggest to merge PR772 into the lastest version.

[tridao]

Can you say the sizes of the inputs that you're benchmarking?
My main concern with PR722 is the potential for deadlock since the threadblocks are synchronizing with each other and I'm not sure if they're guaranteed to scheduled at the same time?
The other concern is more templating which might increase compilation time.

[defei-coder]

Thanks for your reply. I tested on llama-70B and other models. PR722 based on cutlass split-k gemm(https://github.com/NVIDIA/cutlass/blob/main/include/cutlass/gemm/kernel/gemm.h#L300C5-L300C67,https://github.com/NVIDIA/cutlass/blob/main/include/cutlass/semaphore.h). Perhaps cutlass engineer could help review the code.