[new] CUDA error: an illegal memory access was encountered

[Klawens]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Detection, Export

Bug

After successfully exported to yolov5s.engine, I use the code similar to detect.py to get frame results, got CUDA error: an illegal memory access was encountered.
[TensorRT] ERROR: ../rtSafe/cuda/caskConvolutionRunner.cpp (373) - Cask Error in checkCaskExecError<false>: 7 (Cask Convolution execution) [TensorRT] ERROR: FAILED_EXECUTION: std::exception Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/nvidia/alphapose/alphapose/utils/webcam_detector.py", line 131, in frame_preprocess img_det = self.image_detection((img_k, orig_img, im_name, im_dim_list_k)) File "/home/nvidia/alphapose/alphapose/utils/webcam_detector.py", line 140, in image_detection dets = self.detector.images_detection(img, im_dim_list) File "/home/nvidia/alphapose/detector/yolov5_hub_api.py", line 108, in images_detection pred = non_max_suppression(pred, self.conf_thres, self.iou_thres, classes, agnostic_nms, max_det=500) File "/home/nvidia/alphapose/detector/yolov5/utils/general.py", line 685, in non_max_suppression xc = prediction[..., 4] > conf_thres # candidates RuntimeError: CUDA error: an illegal memory access was encountered

Environment

YOLOv5 v6.1, Jetson Xavier NX, CUDA 10.2, cudnn8.0, trt 7.1.3

Minimal Reproducible Example

def load_model(self):
    args = self.detector_opt
    device = select_device(0)
    model = DetectMultiBackend(self.model_weights, device=device,     
    data='/home/nvidia/alphapose/detector/yolov5/data/coco128.yaml')
    stride, names, pt, jit, onnx, engine = model.stride, model.names, model.pt, model.jit, model.onnx, model.engine
    imgsz = check_img_size(self.inp_dim, s=stride)  # check image size
        
    # Half
    self.half &= (pt or jit or onnx or engine) and device.type != 'cpu'  # FP16 supported on limited backends with CUDA
    if pt or jit:
        self.model.model.half() if half else model.model.float()
    self.model = model

def images_detection(self, imgs, orig_dim_list=[]):
    """
    Feed the img data into object detection network and 
    collect bbox w.r.t original image size
    Input: imgs(torch.FloatTensor,(b,3,h,w)): pre-processed mini-batch image input
           orig_dim_list(torch.FloatTensor, (b,(w,h,w,h))): original mini-batch image size
    Output: dets(torch.cuda.FloatTensor,(n,(batch_idx,x1,y1,x2,y2,c,s,idx of cls))): human detection results
    """
    if not self.model:
        self.load_model()

    dets = []
    # [b, c, h, w]
    print(imgs.shape)
    # self.model.warmup(imgsz=(1 if pt else bs, 3, *imgsz), half=half)  # warmup
    dt, seen = [0.0, 0.0, 0.0], 0
    for idx, im in enumerate(imgs):
        dets.append([])
        #t1 = time_sync()
        #im = torch.from_numpy(im).to(device)
        print(im.shape) # should be torch.Size([3, 448, 448]
        im = im.half() if self.half else im.float()  # uint8 to fp16/32
        im /= 255  # 0 - 255 to 0.0 - 1.0
        if len(im.shape) == 3:
            im = im[None]  # expand for batch dim
        #t2 = time_sync()
        #dt[0] += t2 - t1

        # Inference
        visualize = increment_path(save_dir / Path(path).stem, mkdir=True) if self.visualize else False
        pred = self.model(im, augment=False, visualize=visualize)
        #t3 = time_sync()
        #dt[1] += t3 - t2

        # NMS
        agnostic_nms = False
        classes = 0
        pred = non_max_suppression(pred, self.conf_thres, self.iou_thres, classes, agnostic_nms, max_det=500)
        #dt[2] += time_sync() - t3

        for p in pred:
            if p[-1] == 0:
                dets[-1].append([idx, p[0], p[1], p[2], p[3], p[4], p[4], 0])

    
    return dets

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[github-actions]

👋 Hello @Klawens, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@Klawens 👋 hi, thanks for letting us know about this possible problem with YOLOv5 🚀. TensorRT export and inference are working correctly with export.py and detect.py:

python export.py --weights yolov5s.pt --include engine
python detect.py --weights yolov5s.engine

We've created a few short guidelines below to help users provide what we need in order to start investigating a possible problem.

How to create a Minimal, Reproducible Example

When asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as creating a minimum reproducible example. Your code that reproduces the problem should be:

• ✅ Minimal – Use as little code as possible to produce the problem
• ✅ Complete – Provide all parts someone else needs to reproduce the problem
• ✅ Reproducible – Test the code you're about to provide to make sure it reproduces the problem

For Ultralytics to provide assistance your code should also be:

• ✅ Current – Verify that your code is up-to-date with GitHub master, and if necessary git pull or git clone a new copy to ensure your problem has not already been solved in master.
• ✅ Unmodified – Your problem must be reproducible using official YOLOv5 code without changes. Ultralytics does not provide support for custom code ⚠️.

If you believe your problem meets all the above criteria, please close this issue and raise a new one using the 🐛 Bug Report template with a minimum reproducible example to help us better understand and diagnose your problem.

Thank you! 😃

[glenn-jocher]

@Klawens good news 😃! Your original issue may now be fixed ✅ in PR #7560 implementing a TRT PyTorch Hub inference solution by @youngjae-avikus.

To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[meheraj2325]

@glenn-jocher Thanks for fixing it. Literally, I was trying my best to fix this issue for the last 2 weeks. Because I wanted to use the tensorRT engine for inference through torch-hub for a person-detection model.

But I am kinda frustrated 😅. I have just found the solution for this pytorch-hub and tensorRT issue an hour ago. I was too happy after solving the issue by myself. Maybe I wrote hundreds of print statements just to debug it and to find the reason why it was working using detect.py but not using torch-hub. After detecting the problem, i just changed a line and it worked.

Then I came here to give the good news and wanted to create a pull request for this fix. But you already gave the good news.

[glenn-jocher]

@meheraj2325 thank you for your efforts! There are quite a few outstanding TODOs currently on YOLOv5 that you might be able to contribute to: https://github.com/ultralytics/yolov5/issues?q=label%3ATODO+, or if you see any other improvements that might be made please let us know!