Zero-mAP bug: P, R and mAP are all 0.0

[athrunsunny]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I train my custom one class dataset（about 40,000 pictures）use default cfg of yolov5s with the latest version（6.2）,P, R and mAP are all 0，(It's the same after about 100 epochs)however, it is normal to use the previous version（6.1）and it's normal when I train with coco128,I think this may be a bug, but I didn't debug it.

Additional

No response

[github-actions]

👋 Hello @athrunsunny, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[giuseppebrb]

It's a bug already happened the other day but @glenn-jocher fixed it, today reappeared

[pourmand1376]

Have you updated your repo? @glenn-jocher has created a PR for this yesterday.

[giuseppebrb]

In my case, yesterday everything was working again, today bug returned. I clone the repository at every run in my Google Colab notebook.

[glenn-jocher]

@athrunsunny @giuseppebrb @pourmand1376 I'm still seeing this myself, but it's not consistently reproducible. I trained 10 models in a row in Colab just now and the zero-mAP bug randomly appears in 2 out of 10. Importantly all val losses are identical, all single-gpu trainings are now 100% reproducible, so the bug is not in training, it is in the metrics apparently.

I need help debugging this, I don't know what's causing the problem.

# Train 10x YOLOv5s on COCO128 for 3 epochs
for _ in range(10):
  !python train.py --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache

@AyushExel FYI this is happening, don't have a solution yet.

[pourmand1376]

@glenn-jocher
Okay, If you don't know where to start, I suggest using git bisect. This one will find the exact commit which has made the issue.

I will try to solve the problem. However, I don't know any commit which doesn't have this issue. Do you know any?

[glenn-jocher]

@pourmand1376 yes I've been manually bisecting commits, but since it's not 100% reproducible I would only test one training, and the most recent commit that passed the one training I would revert to. This only appeared in the last few days. Some other observations:

• --device cpu trainings always appear to work correctly
• --noplots trainings always appear to work correctly even on GPU
• another user mentioned nan batchnorm values, but this doesn't make sense to me as all val losses are identical for both zero map and normal runs.

[AyushExel]

This seems like a p0 problem. Let's prioritize this.
I've seen similar things happen when some hyper-param is set to a bad value. Was there any commit lately related to hyps?

[glenn-jocher]

@pourmand1376 so it seems like the git bisect approach is correct, but the test must be high statistics, i.e. run 30 trainings with 30 good results and we can be 97% sure there's no bug in that commit.

[glenn-jocher]

@AyushExel no, last hyp changes were months ago. I think this might be related to EMA or running val.py directly from train.py or maybe AMP/FP16 ops between train/val. But the strangest part is that val losses are always identical across all models, regardless of whether map is 0 or correct.

Also another observation is that calling val.py directly always works. The bug is in val during training only. In other words, there is absolutely nothing wrong with training or the models that are being trained, what's wrong is that val.py during training is erroneously reporting zero-mAP.

[glenn-jocher]

Another thing I suspected was the introduction of identical filenames in /classify. This has never happened before in the repo. If that's the case then the bug would appear directly in the v6.2 tag.