Fix tokens count for benchmark

[DrownFish19]

PR types

Bug fixes

PR changes

Others

Description

Set total_effective_tokens=all_tokens * progress_or_epoch.
When training or fine-tuning with a max_step and using Data Parallelism (DP) or Sharding with a factor greater than 1, the calculation of total_effective_tokens may be incorrect.

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.04%. Comparing base (0cc8554) to head (4e867ff).
Report is 231 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #8938      +/-   ##
===========================================
- Coverage    55.05%   55.04%   -0.02%     
===========================================
  Files          635      635              
  Lines        99410    99436      +26     
===========================================
+ Hits         54729    54730       +1     
- Misses       44681    44706      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.