SwiGLU kernel design choice.

[khalil-Hennara]

Hi,

I was looking for the kernel at the unsloth repo, and looking for SwiGLU kernel, I've notice that you reshape the activation matrix, 'e and g' make it 2D matrix, while in the forward pass the matrix shape didn't change (batch_size, Sequance, h_d). I've noticed that the way we acess data the same for both kernel, my question is, is there a reson to reshape the matrix within the backward pass kernel, even thought we didn't reshape the matrix in the forward pass and works fine ?

Another question is about why did you re-compute the result 'h_row = f_row * g_row' and replace DW the gradient with this activation value.

I hope to answer @danielhanchen