Use test.py for custom labelled dataset

[khalidw]

❔Question

Hi! I have a custom dataset of boats (only one class). I have labelled the data myself. I was wondering how can I generate mAP metrics for this data using the pretrained weights (yolov5x.pt).

I understand that test.py can be used to generate mAP metrics but I cannot figure out how can this be done for my own labelled dataset.

Additional context

[glenn-jocher]

@khalidw mAP is automatically computed using test.py after every epoch during training. See Train Custom Data tutorial to get started:

Tutorials

• Train Custom Data  🚀 RECOMMENDED
• Weights & Biases Logging  🌟 NEW
• Multi-GPU Training
• PyTorch Hub  ⭐ NEW
• ONNX and TorchScript Export
• Test-Time Augmentation (TTA)
• Model Ensembling
• Model Pruning/Sparsity
• Hyperparameter Evolution
• Transfer Learning with Frozen Layers  ⭐ NEW
• TensorRT Deployment

[khalidw]

@glenn-jocher Thanks for your response but it seems like my query was unclear.

• I have two models: pretrained and custom trained
• For the custom trained models I have got the mAP which were generated while training
• Now I want to generate mAP for the pretrained model on my custom dataset
• Purpose of this is to compare the performance of custom trained and pretrained model on my custom dataset

I am hoping that custom trained model would have better mAP score than the pre-trained model for my custom dataset

[glenn-jocher]

@khalidw you can use any model you want with test.py by passing it with the --weights argument:
python test.py --data your_data.yaml --weights any_model.pt

If the model was trained on the data, or has intersecting classes with your dataset then you should get some nonzero mAP result.

[khalidw]

@glenn-jocher this works perfectly fine with my custom trained models. But I also want to get mAP for pretrained models so that a comparison can be made.

Although mAP for pretrained models are given, but the provided values are for VOC and COCO dataset. I want to generate mAP for pretrained models on my custom dataset.

When I tried to run the test.py for a pretrained model using my custom dataset (boats) I ran into an error. I wasn't expecting this error as my custom dataset and the pretrained models have an intersecting class (boats).

I added an entry in the localDataset/localDataset.yaml as test: localDataset/test.txt which points to the location of test images

I am sharing both the results for custom trained model (error free) and pretrained model (with an error)

mAP for pretrained yolov5x

!python test.py --weights yolov5x.pt --data localDataset/localDataset.yaml --img 640

Namespace(augment=False, batch_size=32, conf_thres=0.001, data='localDataset/localDataset.yaml', device='', exist_ok=False, img_size=640, iou_thres=0.6, name='exp', project='runs/test', save_conf=False, save_hybrid=False, save_json=False, save_txt=False, single_cls=False, task='val', verbose=False, weights=['yolov5x.pt'])
YOLOv5 v4.0-20-ge8a41e8 torch 1.7.0+cu101 CUDA:0 (Tesla T4, 15109.75MB)

Fusing layers... 
Model Summary: 476 layers, 87730285 parameters, 0 gradients, 218.8 GFLOPS
val: Scanning 'localDataset/labels.cache' for images and labels... 100 found, 0 missing, 0 empty, 0 corrupted: 100% 100/100 [00:00<00:00, 731990.23it/s]
               Class      Images     Targets           P           R      mAP@.5  mAP@.5:.95:   0% 0/4 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "test.py", line 321, in <module>
    save_conf=opt.save_conf,
  File "test.py", line 182, in test
    confusion_matrix.process_batch(pred, torch.cat((labels[:, 0:1], tbox), 1))
  File "/gdrive/My Drive/object_detection/YOLOv5/utils/metrics.py", line 146, in process_batch
    self.matrix[gc, detection_classes[m1[j]]] += 1  # correct
IndexError: index 8 is out of bounds for axis 1 with size 2

mAP for custom trained yolov5x

# yolov5x
!python test.py --weights runs/train/exp3/weights/best.pt --data localDataset/localDataset.yaml --img 640

Namespace(augment=False, batch_size=32, conf_thres=0.001, data='localDataset/localDataset.yaml', device='', exist_ok=False, img_size=640, iou_thres=0.6, name='exp', project='runs/test', save_conf=False, save_hybrid=False, save_json=False, save_txt=False, single_cls=False, task='val', verbose=False, weights=['runs/train/exp3/weights/best.pt'])
YOLOv5 v4.0-20-ge8a41e8 torch 1.7.0+cu101 CUDA:0 (Tesla T4, 15109.75MB)

Fusing layers... 
Model Summary: 476 layers, 87198694 parameters, 0 gradients, 217.1 GFLOPS
val: Scanning 'localDataset/labels.cache' for images and labels... 100 found, 0 missing, 0 empty, 0 corrupted: 100% 100/100 [00:00<00:00, 1061849.11it/s]
               Class      Images     Targets           P           R      mAP@.5  mAP@.5:.95: 100% 4/4 [00:04<00:00,  1.03s/it]
                 all         100         304       0.731       0.947       0.897       0.463
Speed: 19.7/2.4/22.0 ms inference/NMS/total per 640x640 image at batch-size 32
Results saved to runs/test/exp2

[glenn-jocher]

@khalidw you can only test models on datasets with identical classes. You test a COCO trained model on COCO only.

[Akhp888]

why doesnt the sum add up to 1 for Predicted scores in confusion matrix ?

[glenn-jocher]

@Akhp888 the columns are normalized in the confusion matrix, not the rows.

There is also a PR #2114 open with a slightly different confusion matrix implementation that you may want to look at.

[Akhp888]

@Akhp888 the columns are normalized in the confusion matrix, not the rows.

There is also a PR #2114 open with a slightly different confusion matrix implementation that you may want to look at.

Thanks for the reply @glenn-jocher
Taking into consideration about the normalization you mentioned I still find it hard to interpret the Background FN /FP , can you help me with generalising the scores ?
Also I tried with PR #2114 where i see the axis of miscalculation has reversed but still not something which i could relate .

Thanks

[glenn-jocher]

@Akhp888 yeah don't worry, the confusion matrix is certainly pretty confusing. In general everyone is used to seeing classification confusion matrices, which are simpler due to lack of the background class we have here.

I'm not sure about row and column normalization both at the same time. Is that even possible? i.e. is there a linear closed form solution for that?

You can try to test at a few different confidence levels to understand how the confusion matrix works (i.e. 0.001, 0.1, 0.9)

[Akhp888]

@Akhp888 yeah don't worry, the confusion matrix is certainly pretty confusing. In general everyone is used to seeing classification confusion matrices, which are simpler due to lack of the background class we have here.

I'm not sure about row and column normalization both at the same time. Is that even possible? i.e. is there a linear closed form solution for that?

You can try to test at a few different confidence levels to understand how the confusion matrix works (i.e. 0.001, 0.1, 0.9)

I was trying to figure out any way by which i could have same normalization for ground truth values and predicted when i noticed that the inbuilt xyxy2xywh and xywh2xyxy giving me different value than what is expected.
eg : for xyxy = [5994.9658203125, 1397.2547607421875, 6290.80908203125, 1770.0487060546875]
with gn = [32579,2048,32579,2048]
i got xywh = [0.1885536015033722, 0.7732674479484558, 0.009080796502530575, 0.18202829360961914]

where as by manual calculation i was supposed to get [0.1840131931708309, 0.6822533011436462, 0.009080796271179288, 0.18202829360961914]
notice the difference in y value ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[edervishaj]

@khalidw you can only test models on datasets with identical classes. You test a COCO trained model on COCO only.

So does this mean that yolov5 cannot be used for unseen data from another totally different dataset?