the MAP of verification suddenly becomes 0 during training

[yrn1126]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

During the training of my own dataset, everything was normal. But at the 80th epoch, the mAP of the verification became 0 suddenly . After 10 epochs, it becomes normal again, but the mAP was much lower than before. I feel confused about this question.

Additional

No response

[josebenitezg]

Hi @yrn1126 I have the same problem

[glenn-jocher]

@josebenitezg @yrn1126 your losses are increasing.

[josebenitezg]

hi @glenn-jocher
Actually, first are decreasing but then are incresing again, and recover again.
I try with yolov5s6 model with same dataset and I don't have this problem.

[glenn-jocher]

@josebenitezg you may need to use less aggressive hyps or a longer warmup etc. to increase stability.

[josebenitezg]

@glenn-jocher I put the default hyps but still same error with large model (even with the last commit)
As you can see here, I train yolov5s6 with no problem (orange one). Same dataset with large suddenly becomes 0

hyperparameters: lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
Training on EC2 Instance p3.2xlarge Instance

[glenn-jocher]

@josebenitezg perhaps related to warmup bug fix #8356 merged today. You should git pull and train again.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!