Many errors occur during the training process

[yongliang-wu]

Hi,
Thanks for your excellent work. During training I encountered lots of errors and some garbled output in the samples— is this expected? On 8 A100 GPUs it shows that training 444 steps will take 40 hours, roughly 5 minutes per step. Is that a normal training speed?

Qwen2.5-Math-1.5B-raft-plusplus-numina_math-n4.log

[WeiXiongUST]

Hi, the error outputs are from math-verify and are expected. Actually, lots of people have asked math-verify team if they could turn off these messages but so far no progress...

The speed is also expected.

[yongliang-wu]

Thanks for you reply. Btw, can you share the hyper-parameter settings? As you mentioned in your paper, it will be found at appendix which seems hasn't uploaded to the arxiv. Many thanks for your help.