How to compile Yolov5 for AWS Neuron

[fishuke]

I faced with lack of documentation about this and i wanted to share how i compiled and infered yolov5s6.

Prerequisites:

Linux (not working with mac) preferably aws inf1 instance

Create a Python 3.7.10 venv.
Follow here
Install yolov requirements
pip install "torch-neuron==1.10.2.*" "neuron-cc[tensorflow]" "protobuf<4" torchvision==0.11.3 --extra-index-url https://pip.repos.neuron.amazonaws.com

Compiler:

import torch
import torch_neuron

model = torch.hub.load('ultralytics/yolov5', 'yolov5s6')  # whatever version do you need

for m in model.modules():
    if hasattr(m, 'inplace'):
        m.inplace = False

fake_image = torch.zeros([1, 3, 640, 640], dtype=torch.float32) # customize size here 640x640 is common

try:
    torch.neuron.analyze_model(model, example_inputs=[fake_image])
except Exception:
    torch.neuron.analyze_model(model, example_inputs=[fake_image])

model_neuron = torch.neuron.trace(model,
                                example_inputs=[fake_image])

## Export to saved model
model_neuron.save("neuron_yolov5s6.pt")

Inference:

import cv2
import torch
import numpy as np
import torch.neuron
from util_yolo import non_max_suppression

im = cv2.imread('img.jpg')
# img0 = im.copy()
im = cv2.resize(im, (640, 640), interpolation = cv2.INTER_AREA)
img0 = im.copy()
# Convert
im = im.transpose((2, 0, 1))[::-1]  # HWC to CHW, BGR to RGB
im = np.ascontiguousarray(im)
# Convert into torch
im = torch.from_numpy(im)
im = im.float()  # uint8 to fp16/32
im /= 255  # 0 - 255 to 0.0 - 1.0
if len(im.shape) == 3:
    im = im[None]  # expand for batch dim

# Load the compiled model
model = torch.jit.load('neuron_yolov5s6.pt')

CLASSES = ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light',
        'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
        'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee',
        'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard',
        'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple',
        'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch',
        'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard',
        'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors',
        'teddy bear', 'hair drier', 'toothbrush']

# Inference
pred = model(im)
pred = non_max_suppression(pred) #nms function used same as yolov5 detect.py

#Process predictions
for i, det in enumerate(pred):  # per image
    im0 = img0.copy()
    color=(30, 30, 30)
    txt_color=(255, 255, 255)
    h_size, w_size = im.shape[-2:]
    print(h_size, w_size)
    lw = max(round(sum(im.shape) / 2 * 0.003), 2)
    # cv2.rectangle(im0, (10,10), (200,200), (0,0,0))
    if len(det):
        # Write results
        for *xyxy, conf, cls in reversed(det):
            c = int(cls)  # integer class
            label = f'{CLASSES[c]} {conf:.2f}'
            print(label)
            #label = "human"
            box = xyxy
            # p1, p2 = (int(box[0]* w_size), int(box[1]* h_size)), (int(box[2]* w_size), int(box[3]* h_size))
            p1, p2 = (int(box[0]), int(box[1])), (int(box[2]), int(box[3]))
            cv2.rectangle(im0, p1, p2, color, thickness=lw, lineType=cv2.LINE_AA)
            print(f'p1={p1}, p2={p2}, box={xyxy}')
            tf = max(lw - 1, 1)  # font thickness
            w, h = cv2.getTextSize(label, 0, fontScale=lw / 3, thickness=tf)[0]  # text width, height
            outside = p1[1] - h - 3 >= 0  # label fits outside box
            p2 = p1[0] + w, p1[1] - h - 3 if outside else p1[1] + h + 3
            cv2.rectangle(im0, p1, p2, color, -1, cv2.LINE_AA)  # filled
            cv2.putText(im0,
                        label, (p1[0], p1[1] - 2 if outside else p1[1] + h + 2),
                        0,
                        lw / 3,
                        txt_color,
                        thickness=tf,
                        lineType=cv2.LINE_AA)
    # Save results (image with detections)
    status = cv2.imwrite('out.jpg', im0)

[github-actions]

👋 Hello @fishuke, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[fishuke]

#7739

[vitorStein]

Running the inference.py I got the following error:

RuntimeError: 
    Incorrect tensor shape at input tensor #0: received 1 3 384 640, expected 1 3 640 640.

To correct the error we must resize the image to the same size as the fake_image used in the compiler.py

# inference.py 
im = cv2.resize(im, (640, 640), interpolation = cv2.INTER_AREA)

[fishuke]

@vitorStein thanks for the correction. I updated my post 👍

[ianhco]

I've previously sucessfully compiled artifacts, but lately suddenly an error came out of the blue preventing me from compiling and I am not even sure where it comes from 🤷
:List inputs to traced functions must have consistent element type. Found Tensor and List[Tensor]
it happens on the line of torch.neuron.analyze_model(model, example_inputs=[fake_image]) inside except

[ianhco]

@fishuke I have debugged, and found out that the latest branch/tag on master (v6.2) is breaking the compilation, by specifying the branch or tag of previous working versions would fix this issue:

model = torch.hub.load('ultralytics/yolov5:v6.1', 'yolov5s6')

I want the new improvements of 6.2 but can't now 😞

[glenn-jocher]

@paaksing I'm not sure where this error is originating but if you can debug and find a solution please help us by submitting a PR to master. Thank you!

[ianhco]

@glenn-jocher I opened #9341, the issue seems to not just about torch.neuron.trace, also torch.jit.trace, unfortunately I don't have much ground on the internals of yolo so it would be hard for me to debug, for now I will just get going with 6.1

[glenn-jocher]

@paaksing I don't understand the cause, as v6.1 and v6.2 detection models are identical. The main change in v6.2 is the addition of classification models.

[ianhco]

Issue may now be fixed with #9341 and #9363, will close this issue once I confirm that torch.neuron.trace is compiling as expected.

[glenn-jocher]

@iann838 Thank you for your contributions! We appreciate your diligence in resolving this issue. Let us know if you encounter any further difficulties.