On the Implementation of REINFORCE_rej

[SihengLi99]

Hi,

Congratulations on your excellent work!
I have been carefully reading your paper and studying the code, and I greatly appreciate your contributions.

While reviewing the implementation of REINFORCE_rej, I was surprised to find that the rewards for positive and negative samples are set to 0 and 1, instead of -1 and 1 as stated in the paper. This appears to result in REINFORCE_rej effectively ignoring all negative samples during training. I find this quite confusing and would greatly appreciate your clarification on this point.

Best regards,
Siheng

[Alex-HaochenLi]

I notice the same problem.

I check the code under /reward_score, except math_dapo.py, all other reward calculation functions assign a reward of 0 for a wrong response.

I think this works fine with RAFT++, because RAFT++ does not use negative responses for training. For wrong responses, their advantages are 0, and during optimization, the loss for tokens in wrong responses is 0.

But assigning a reward of 0 for wrong responses seems not to be true for other RL algorithms. Let's see how the authors would reply.

[hendrydong]

Hi, when we implement it, we changed the reward definition to 1 and -1, so that the current codes work well.
You can redefine the reward.

We will also look into this and make it more general.