Reinforce or Reinforce++? #8
Description
Hi, authors,
it seems that in your code, you use reinforce++ instead of the traditional reinforce algorithm? However, when I read your paper, it looks like you are using the traditional reinforce algorithm, which confuses me.
I noticed this problem because your experimental results seem to be in contradiction with paper "The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning". In the latter, the authors show that using the traditional reinforce algorithm combined with negative samples can yield very good results, while in the figure 4 of your paper, the results of "Reinforce + Remove all correct" are worse. Can you tell me what the difference is?
Besides, could you tell me where you made the modification to the "reward" function? I remember that for the correct and incorrect samples, verl returns 1 and 0 instead of 1 and -1 (Please forgive me that I'm not very familiar with this framework).