[QUESTION]PipeOffload Setting

[Lhongpei]

How should I set arg "--offload-time"? Should I calculate it maually or is there a way to calculate automatically in code?

And can you explain more about the what is a chunk in pipeline parallelism?

[ufotalent]

HI @Lhongpei , thanks for the interest in our work. Currently it can be manually calculated, for our experiments on A100, we used the following formula 1, whith B_c = $220* 10^4$ (220TFLOP/s) and B_o = $15 * 10^3$ (15G/s)

If you're using other hardware, you can get a brief estimation of the compute/PCI-E bandwidth.

We understand that it's a bit tricky to set the parameter, we'll support automatic setting soon

'Chunk' is first introduced in Megatron-LM, known as the 'Interleaved 1F1B' schedule (see figure 4 in the Megatron paper). In this kind of settings, multiple chunks can be on the same device. One simple example can be considering a model with 16 layers put on 4 devices, in naive PP, device $i$ has one chunk of 4 layers $4i$, $4i+1$, $4i+2$, $4i+3$. If we allow each device to have two chunks (Interleaved 1F1B), then device $i$ has a chunk of 2 layers $2i$, $2i+1$ and another chunk of two layers $2i+8$ and $2i+9$