AutoBatch: CUDA anomaly detected

[alexk-ede]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

So I'm testing the autobatch feature which is pretty cool.
It seemed to work fine last week, but this week for whatever reason (maybe bc it's Monday, who knows ...) I'm having issues with it.

I'm running the yolov5s (latest git checkout ofc) and getting this (when using --batch -1)
Dataset is a slice from COCO

AutoBatch: Computing optimal batch size for --imgsz 416
AutoBatch: CUDA:0 (NVIDIA GeForce RTX 3070) 7.79G total, 2.20G reserved, 0.05G allocated, 5.54G free
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
     7027720       6.744         2.414         27.87         35.49        (1, 3, 416, 416)                    list
     7027720       13.49         1.378         23.52         50.14        (2, 3, 416, 416)                    list
     7027720       26.98         1.380          23.8         56.75        (4, 3, 416, 416)                    list
     7027720       53.95         0.648         22.86         71.21        (8, 3, 416, 416)                    list
     7027720       107.9         1.330         26.38         91.88       (16, 3, 416, 416)                    list
AutoBatch: WARNING: ⚠️ CUDA anomaly detected, recommend restart environment and retry command.
AutoBatch: Using batch-size 16 for CUDA:0 0.96G/7.79G (12%) ✅

Meanwhile, the nvtop output is this before running the train.py
So there isn't really anything in the GPU memory.

Device 0 [NVIDIA GeForce RTX 3070] PCIe GEN 1@16x RX: 0.000 KiB/s TX: 0.000 KiB/s
 GPU 210MHz  MEM 405MHz  TEMP  53°C FAN  38% POW  19 / 220 W
 GPU[                                 0%] MEM[|                   0.208Gi/8.000Gi]

I am unsure about this from AutoBatch

7.79G total, 2.20G reserved, 0.05G allocated, 5.54G free

The 2.20G reserved is weird, because I stopped everything (including gdm3), so nothing is running on the GPU.
(besides the training process later).

And I can easily set batch to 80 and it works fine:

 Device 0 [NVIDIA GeForce RTX 3070] PCIe GEN 3@16x RX: 30.27 MiB/s TX: 8.789 MiB/s
 GPU 1905MHz MEM 6800MHz TEMP  68°C FAN  63% POW 199 / 220 W
 GPU[||||||||||||||||||||||||||||||||90%] MEM[||||||||||||||||||||7.319Gi/8.000Gi]

    PID USER DEV    TYPE  GPU        GPU MEM    CPU  HOST MEM Command
   6404 user   0 Compute  91%   7237MiB  88%   105%  14616MiB python train.py --img 416 --batch 80 --epochs 400  --cache --weights yolov5s.pt --data ...

I obviously did the recommended restart environment and even restarted the machine. Autobatch still complained about around 2.20G reserved

Any ideas how I can investigate this ?

My guess is, the 2.2GB do mess up the interpolation for autobatch because the GPU_mem (GB) column doesn't make much sense.

  GPU_mem (GB)       input 
   2.414       (1, 3, 416, 416)
   1.378       (2, 3, 416, 416)
   1.380       (4, 3, 416, 416)
   0.648       (8, 3, 416, 416) 
   1.330       (16, 3, 416, 416)

Additional

• Maybe the issue title should be changed to AutoBatch: CUDA anomaly detected

• some additional system info

Ubuntu 22.04.1 LTS
Kernel Linux 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

nvidia-smi
Mon Sep  5 16:22:01 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |

• Update:
  During training, it shows me usage around

      Epoch    GPU_mem   ...
    112/399      5.79G

So not sure where the rest went (aka the difference to the 7.2GB in nvtop) ...

[github-actions]

👋 Hello @alexk-ede, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[Denizzje]

Hi and happy monday to you.

I use autobatch quite frequently and also just updated my local yolov5 today so I took a look.

GTX 1080 (8GB), PyTorch 1.12, Nvidia driver 515, Cuda 11.7, Fedora 36

When running a similar training like yours, so with input size 416 on Coco128 (I assume you mean that with slice of coco), I too get the warning.

Transferred 481/481 items from yolov5m.pt
AMP: checks passed ✅
AutoBatch: Computing optimal batch size for --imgsz 416
AutoBatch: CUDA:0 (NVIDIA GeForce GTX 1080) 7.92G total, 2.53G reserved, 0.16G allocated, 5.24G free
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
    21190557        20.8         2.816         34.12         119.1        (1, 3, 416, 416)                    list
    21190557       41.61         2.852         31.86         149.8        (2, 3, 416, 416)                    list
    21190557       83.21         2.842         30.95         173.2        (4, 3, 416, 416)                    list
    21190557       166.4         2.772         46.22         201.2        (8, 3, 416, 416)                    list
    21190557       332.9         2.741         82.16         318.4       (16, 3, 416, 416)                    list
AutoBatch: WARNING: ⚠️ CUDA anomaly detected, recommend restart environment and retry command.
AutoBatch: Using batch-size 16 for CUDA:0 2.74G/7.92G (35%) ✅
optimizer: SGD(lr=0.01) with parameter groups 79 weight(decay=0.0), 82 weight(decay=0.0005), 82 bias
train: Scanning '/home/xxx/ai_dev/datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]                                                      
val: Scanning '/home/xxx/ai_dev/datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]                                                        

AutoAnchor: 4.18 anchors/target, 0.977 Best Possible Recall (BPR). Anchors are a poor fit to dataset ⚠️, attempting to improve...
AutoAnchor: WARNING: Extremely small objects found: 16 of 929 labels are < 3 pixels in size
AutoAnchor: Running kmeans for 9 anchors on 927 points...
AutoAnchor: Evolving anchors with Genetic Algorithm: fitness = 0.6699: 100%|██████████| 1000/1000 [00:00<00:00, 1421.07it/s]                                                                                                                 
AutoAnchor: thr=0.25: 0.9935 best possible recall, 3.75 anchors past thr
AutoAnchor: n=9, img_size=416, metric_all=0.263/0.670-mean/best, past_thr=0.477-mean: 6,9, 16,14, 21,35, 55,47, 70,94, 80,188, 190,139, 216,249, 388,283
AutoAnchor: Done ✅ (optional: update model *.yaml to use these anchors in the future)
Plotting labels to runs/train/exp7/labels.jpg... 
Image sizes 416 train, 416 val
Using 8 dataloader workers
Logging results to runs/train/exp7
Starting training for 300 epochs...

When not inputting an input size and 640 is used, I do not get the CUDA enviornment warning.

AutoBatch: Computing optimal batch size for --imgsz 640
AutoBatch: CUDA:0 (NVIDIA GeForce GTX 1080) 7.92G total, 2.53G reserved, 0.16G allocated, 5.24G free
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
    21190557       49.24         2.802         46.01         252.1        (1, 3, 640, 640)                    list
    21190557       98.48         2.751         36.03           286        (2, 3, 640, 640)                    list
    21190557         197         2.724         50.43         334.9        (4, 3, 640, 640)                    list
    21190557       393.9         2.810         94.75         416.9        (8, 3, 640, 640)                    list
    21190557       787.8         5.492         180.2         560.2       (16, 3, 640, 640)                    list
AutoBatch: Using batch-size 11 for CUDA:0 4.18G/7.92G (53%) ✅
optimizer: SGD(lr=0.01) with parameter groups 79 weight(decay=0.0), 82 weight(decay=0.000515625), 82 bias
train: Scanning '/home/xxx/ai_dev/datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]                                                                                                                                      
val: Scanning '/home/xxx/ai_dev/datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]                                                                                                                                        

AutoAnchor: 4.27 anchors/target, 0.994 Best Possible Recall (BPR). Current anchors are a good fit to dataset ✅
Plotting labels to runs/train/exp6/labels.jpg... 
Image sizes 640 train, 640 val
Using 8 dataloader workers
Logging results to runs/train/exp6
Starting training for 300 epochs...

I too have this random chunk of 2.53 in my GPU memory. I do not know what it is neither, but it does not match with my usage in nvitop before training start (around 500-600mb, with gnome desktop and xorg on). Checking back from autobatch trainings on the initial release of v6.2 , I do see:

�[34m�[1mAutoBatch: �[0mComputing optimal batch size for --imgsz 1280
�[34m�[1mAutoBatch: �[0mCUDA:0 (NVIDIA A100-SXM-80GB) 79.35G total, 0.11G reserved, 0.10G allocated, 79.14G free
1660813541500 os-nvme-hbwt3yi6-2a100-44v-fin1 info       Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
    12403204       65.97         0.872         29.57         20.63      (1, 3, 1280, 1280)                    list
    12403204       131.9         1.629         26.51         22.46      (2, 3, 1280, 1280)                    list
    12403204       263.9         3.181         31.44         28.91      (4, 3, 1280, 1280)                    list
1660813542584 os-nvme-hbwt3yi6-2a100-44v-fin1 info     12403204       527.8         5.981         33.11         43.87      (8, 3, 1280, 1280)                    list
    12403204        1056        12.092         54.42         79.66     (16, 3, 1280, 1280)                    list
1660813543232 os-nvme-hbwt3yi6-2a100-44v-fin1 error �[34m�[1mAutoBatch: �[0mUsing batch-size 95 for CUDA:0 70.93G/79.35G (89%) ✅
�[34m�[1moptimizer:�[0m SGD(lr=0.01) with parameter groups 75 weight(decay=0.0), 79 weight(decay=0.0007421875), 79 bias

[alexk-ede]

Hi @Denizzje and a happy start of the week to you, too.
The monday wasn't so bad actually ;)

I'll add another data point today.
This time yolov5m

Transferred 475/481 items from yolov5m.pt
AMP: checks passed ✅
AutoBatch: Computing optimal batch size for --imgsz 640
AutoBatch: CUDA:0 (NVIDIA GeForce RTX 3070) 7.79G total, 2.52G reserved, 0.16G allocated, 5.11G free
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
    20879400       48.25         2.798         33.85         180.3        (1, 3, 640, 640)                    list
    20879400       96.49         2.749         14.16         186.3        (2, 3, 640, 640)                    list
    20879400         193         2.718         20.15         218.2        (4, 3, 640, 640)                    list
    20879400         386         2.720            37         282.9        (8, 3, 640, 640)                    list
    20879400       771.9         5.207         71.34         334.4       (16, 3, 640, 640)                    list
AutoBatch: Using batch-size 11 for CUDA:0 4.01G/7.79G (51%) ✅

But manually a batch of 16 is fine.

So yeah, looks like the 2.52G reserved do interfere, distort the testing for the size and then make the interpolation invalid.
Maybe it'd be better to go by the 0.16G allocated instead, because that is also what nvtop is basically showing.

Instead of just saying anomaly detected it'd be also useful to hint below, that the initial VRAM usage/reserved is suspiciously high.

Update:
I forgot to mention, looks that for your img 640 test, despite no warning, the autobatch result is still not optimal either, as the first 4 results are all around 2.8GB

AutoBatch: Computing optimal batch size for --imgsz 640
AutoBatch: CUDA:0 (NVIDIA GeForce GTX 1080) 7.92G total, 2.53G reserved, 0.16G allocated, 5.24G free
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
    21190557       49.24         2.802         46.01         252.1        (1, 3, 640, 640)                    list
    21190557       98.48         2.751         36.03           286        (2, 3, 640, 640)                    list
    21190557         197         2.724         50.43         334.9        (4, 3, 640, 640)                    list
    21190557       393.9         2.810         94.75         416.9        (8, 3, 640, 640)                    list
    21190557       787.8         5.492         180.2         560.2       (16, 3, 640, 640)                    list
AutoBatch: Using batch-size 11 for CUDA:0 4.18G/7.92G (53%) ✅

[glenn-jocher]

@alexk-ede AutoBatch may produce inaccurate results under certain circumstances, i.e. when previous trainings are in progress or have terminated early or not all CUDA memory has been released. If you find ways to improve please let us know, the relevant code is here:
https://github.com/ultralytics/yolov5/blob/master/utils/autobatch.py

[alexk-ede]

Yes, I saw that file after I was investigating where the warning message came from. That's where I learned about the interpolation, too.

i.e. when previous trainings are in progress or have terminated early or not all CUDA memory has been released.

I could understand that, if there was something using the GPU before, then that may be plausible.

But as I said, this is a completely fresh boot and fresh start of the environment. Nothing was run before, and obviously no trainings in progress, as it says 0.16G allocated.
I still have absolutely no clue, where this 2.53G reserved are coming from (and what it means).
Sure, there is some documentation, but not quite helpful.
https://pytorch.org/docs/stable/generated/torch.cuda.memory_reserved.html

This looks like it makes more sense
https://pytorch.org/docs/stable/generated/torch.cuda.memory_allocated.html

I'd obviously prefer to have a command that gives me the same output as nvtop.

I'll try later to use it here with memory_allocated instead

yolov5/utils/torch_utils.py

Line 189 in 1aea74c

mem = torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0 # (GB)

as it's called by autobatch here

yolov5/utils/autobatch.py

Line 51 in 1aea74c

results = profile(img, model, n=3, device=device)

[alexk-ede]

I decided I'll test what will happen, when I'll run this during a training session that already uses most of the gpu memory.

def autobatch(model, imgsz=640, fraction=0.8, batch_size=16):
    # Automatically estimate best batch size to use `fraction` of available CUDA memory
    # Usage:
    #     import torch
    #     from utils.autobatch import autobatch
    #     model = torch.hub.load('ultralytics/yolov5', 'yolov5s', autoshape=False)
    #     print(autobatch(model))

Turns out it fails on the first try and the results list doesn't get initialized at all.

In [4]: print(autobatch(model))
AutoBatch: Computing optimal batch size for --imgsz 640
AutoBatch: CUDA:0 (NVIDIA GeForce RTX 3070) 7.79G total, 0.04G reserved, 0.03G allocated, 7.73G free
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 7.79 GiB total capacity; 37.38 MiB already allocated; 6.06 MiB free; 56.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 7.79 GiB total capacity; 46.76 MiB already allocated; 6.06 MiB free; 56.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
AutoBatch: CUDA out of memory. Tried to allocate 38.00 MiB (GPU 0; 7.79 GiB total capacity; 28.01 MiB already allocated; 26.06 MiB free; 36.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 print(autobatch(model))

File ~/repo/yolov5/utils/autobatch.py:56, in autobatch(model, imgsz, fraction, batch_size)
     53     LOGGER.warning(f'{prefix}{e}')
     55 # Fit a solution
---> 56 y = [x[2] for x in results if x]  # memory [2]
     57 p = np.polyfit(batch_sizes[:len(y)], y, deg=1)  # first degree polynomial fit
     58 b = int((f * fraction - p[1]) / p[0])  # y intercept (optimal batch size)

UnboundLocalError: local variable 'results' referenced before assignment

But in this case 0.03G allocated is also completely wrong, because the real usage is
GPU[||||||||||||||||||||||||||||||||92%] MEM[||||||||||||||||||||7.975Gi/8.000Gi]

So as planned, I'll try memory_allocated later instead, but it also yields weird results.
Now I see where the main problem is ...

[alexk-ede]

This is quite weird, I just quickly tested this demo code while a training is running and using 6.6GB VRAM.

import torch

print(torch.__version__)
my_tensor = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.float32, device="cpu")
print(my_tensor)
torch.cuda.is_available()


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print('Using device:', device)
print()

print("torch.cuda.is_available()")
print(torch.cuda.is_available())

#Additional Info when using cuda
if device.type == 'cuda':
    print("torch.cuda.current_device()")
    print(torch.cuda.current_device())

    print("torch.cuda.device(0)")
    print(torch.cuda.device(0))

    print("torch.cuda.get_device_name(0)")
    print(torch.cuda.get_device_name(0))
    
    print()
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:   ', round(torch.cuda.memory_reserved(0)/1024**3,1), 'GB')

instead I only get now

<torch.cuda.device object at 0x7f24a6bb09d0>
torch.cuda.get_device_name(0)
NVIDIA GeForce RTX 3070

Memory Usage:
Allocated: 0.0 GB
Cached:    0.0 GB

So I don't know how to fix that with only using the interface that torch provides.

The command nvidia-smi --query-gpu=memory.used --format=csv,nounits,noheader outputs the exact current usage that is also seen in nvtop.
So sure, one could add some functions like these to extract the GPU memory usage, but it's also not a very clean solution https://www.programcreek.com/python/?CodeExample=get+gpu+memory

[Denizzje]

There does seem to be something very wrong with the auto batch size at the moment. I believe it started after this "CUDA Anomaly" detected was implemented, though I did not do much trainings after a big batch right after the release of 6.2.

This time, I tried with latest YoloV5 from master, PyTorch 1.12, Ubuntu 20.04, Python 3.8 with Nvidia drivers 515 and CUDA 11.7 with an A100 80GB SXM GPU. The dataset is my regular dataset of ~40k training pictures this time, so not Coco1128. It spits out the CUDA anomaly warning and then proceeds with a batch size of 16...

AMP: checks passed ✅
AutoBatch: Computing optimal batch size for --imgsz 640
AutoBatch: CUDA:0 (NVIDIA A100-SXM4-80GB) 79.21G total, 4.85G reserved, 0.16G allocated, 74.20G free

    2022-09-16 15:14:19            Params GFLOPs GPU_mem (GB) forward (ms) backward (ms) input output

    2022-09-16 15:14:20            20964261 48.52 5.283 101 133.7 (1, 3, 640, 640) list

    2022-09-16 15:14:22            20964261 97.03 5.232 22.21 245.8 (2, 3, 640, 640) list

    2022-09-16 15:14:23            20964261 194.1 5.203 18.63 158.6 (4, 3, 640, 640) list

    2022-09-16 15:14:25            20964261 388.1 5.205 17.41 258.5 (8, 3, 640, 640) list

    2022-09-16 15:14:27            20964261 776.3 5.203 23.9 318.2 (16, 3, 640, 640) list

    2022-09-16 15:14:27     

AutoBatch: WARNING: ⚠️ CUDA anomaly detected, recommend restart environment and retry command.

AutoBatch: Using batch-size 16 for CUDA:0 5.19G/79.21G (7%) ✅

optimizer: SGD(lr=0.01) with parameter groups 79 weight(decay=0.0), 82 weight(decay=0.0005), 82 bias

For reference, this was an earlier training with the same machine but with PyTorch 1.10, Cuda 11.3 on YoloV5 release 6.2 (not master from that time), with an earlier version of the same dataset (size is roughly the same though).

AMP: checks passed ✅
AutoBatch: Computing optimal batch size for --imgsz 640
AutoBatch: CUDA:0 (NVIDIA A100-SXM-80GB) 79.35G total, 0.29G reserved, 0.27G allocated, 78.78G free
2022-08-18 08:26:33
      Params      GFLOPs  GPU_mem (GB)  forward (ms) backward (ms)                   input                  output
    35397156       49.54         0.698         40.71         24.46        (1, 3, 640, 640)                    list
    35397156       99.07         1.028         34.32         23.26        (2, 3, 640, 640)                    list
    35397156       198.1         1.707         39.83         25.03        (4, 3, 640, 640)                    list
2022-08-18 08:26:34
    35397156       396.3         3.108         36.23         27.72        (8, 3, 640, 640)                    list
    35397156       792.6         5.895         46.38         39.48       (16, 3, 640, 640)                    list
2022-08-18 08:26:34
AutoBatch: Using batch-size 203 for CUDA:0 70.82G/79.35G (89%) ✅
optimizer: SGD(lr=0.01) with parameter groups 103 weight(decay=0.0), 107 weight(decay=0.0015859375), 107 bias

After my fixed size training run (batch size 128) is finished I will try to redo the autobatch on Yolov5 release 6.2. If this 80GB card is convinced there as well that it can only fit a batch size of 16 in its memory, then the cause is somewhere else. I am curious to see what happens if I retry with PyTorch 1.10 with latest master code.

[glenn-jocher]

@Denizzje yes I'm able to reproduce in Colab. Something is not correct. I'll add a TODO to investigate.

[glenn-jocher]

@Denizzje traced to 5d4787b