[FEATURE] support shuffle partition data write to multiple shuffle servers

[xumanbu]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

Currently, data from one partition can only be written to a designated shuffle server.

In some scenarios, such as when a large partition occupies a significant amount of shuffle server disk space, this can lead to server crashes in extreme cases.

Based on #825, we could support writing one partition data to multiple shuffle servers.

Motivation

https://docs.google.com/document/d/1FRGP82R7JBnr3IjmF8PQ6qWP9GnKQsk-N1OWk_f1kSY

Describe the solution

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[xumanbu]

When to trigger the next node write?

Two strategies I propose are:

• Based on the application quota of a shuffle server.
• Based on the shuffle server's health status.

@jerqi @zuston @leixm

[zuston]

+1 for that.

I hope the trigger strategies could be designed as the pluggable mechansim

[connorlwilkes]

How would this differ from rss.data.replica.write

[jerqi]

When to trigger the next node write?

Two strategies I propose are:

• Based on the application quota of a shuffle server.
• Based on the shuffle server's health status.

@jerqi @zuston @leixm

+1 for this proposal.

[jerqi]

How would this differ from rss.data.replica.write

This may only write one replica but the replica can be stored in different servers.

[xumanbu]

@jerqi @zuston added a dataflow diagram in describe the solution for implementation plan.

[zuston]

@jerqi @zuston added a dataflow diagram in describe the solution for implementation plan.

Looks good.

[jerqi]

@jerqi @zuston added a dataflow diagram in describe the solution for implementation plan.

You would better give us a google document like https://docs.google.com/document/d/1p1PksBN2LJ-OtGEHvdyEuH9b1Mv1aD_exMPl4TNaTs0/edit#heading=h.c7fauf9cicbi

[xumanbu]

@jerqi @zuston added a dataflow diagram in describe the solution for implementation plan.

You would better give us a google document like https://docs.google.com/document/d/1p1PksBN2LJ-OtGEHvdyEuH9b1Mv1aD_exMPl4TNaTs0/edit#heading=h.c7fauf9cicbi

OK. added a google doc https://docs.google.com/document/d/1FRGP82R7JBnr3IjmF8PQ6qWP9GnKQsk-N1OWk_f1kSY

[zuston]

LGTM. Please go ahead.