Convert Yolov5 to IR Model in Openvino

[aseprohman]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello @glenn-jocher et all,

Has anyone ever done the conversion of yolov5 models to IR models in openvino? maybe there is a tutorial i can learn? I want to try deploying yolov5 on Intel NCS2/VPU hardware devices.
thanks

Additional

No response

[glenn-jocher]

@aseprohman what is an IR model?

[besmaguesmi]

Hi, @aseprohman I've done the same work! how can I help?

[aseprohman]

@aseprohman what is an IR model?

@glenn-jocher IR is short from Intermediate Representative model that is the OpenVino model format for inferencing model into targetted devices

[aseprohman]

Hi, @aseprohman I've done the same work! how can I help?

hi @besmaguesmi, can you share to me about model conversion technique from .pt or onnx file to IR Models ? how to pass any parameter like input_shape etc ? if I train yolov5m.pt model with parameter like this :
batch: 32
imgsz: 416
class: 1
thanks

[besmaguesmi]

Hi, @aseprohman first you have to convert your model from PyTorch to the onnx format, then to the IR format.

1. create virtual env with Python 3.6
2. Clone yolov5 repo
3. python export.py –weights model.pt –img 640 –batch 1

`# YOLOv5 🚀 by Ultralytics, GPL-3.0 license
"""
Export a PyTorch model to TorchScript, ONNX, CoreML formats

Usage:
    $ python path/to/export.py --weights yolov5s.pt --img 640 --batch 1
"""

import argparse
import sys
import time
from pathlib import Path

import torch
import torch.nn as nn
from torch.utils.mobile_optimizer import optimize_for_mobile

FILE = Path(__file__).absolute()
sys.path.append(FILE.parents[0].as_posix())  # add yolov5/ to path
sys.path.insert(0, './yolov5')
from models.common import Conv
from models.yolo import Detect
from models.experimental import attempt_load
from utils.activations import Hardswish, SiLU
from utils.general import colorstr, check_img_size, check_requirements, file_size, set_logging
from utils.torch_utils import select_device


def export_torchscript(model, img, file, optimize):
    # TorchScript model export
    prefix = colorstr('TorchScript:')
    try:
        print(f'\n{prefix} starting export with torch {torch.__version__}...')
        f = file.with_suffix('.torchscript.pt')
        ts = torch.jit.trace(model, img, strict=False)
        (optimize_for_mobile(ts) if optimize else ts).save(f)
        print(f'{prefix} export success, saved as {f} ({file_size(f):.1f} MB)')
        return ts
    except Exception as e:
        print(f'{prefix} export failure: {e}')


def export_onnx(model, img, file, opset, train, dynamic, simplify):
    # ONNX model export
    prefix = colorstr('ONNX:')
    try:
        check_requirements(('onnx', 'onnx-simplifier'))
        import onnx

        print(f'\n{prefix} starting export with onnx {onnx.__version__}...')
        f = file.with_suffix('.onnx')
        torch.onnx.export(model, img, f, verbose=False, opset_version=opset,
                          training=torch.onnx.TrainingMode.TRAINING if train else torch.onnx.TrainingMode.EVAL,
                          do_constant_folding=not train,
                          input_names=['images'],
                          output_names=['output'],
                          dynamic_axes={'images': {0: 'batch', 2: 'height', 3: 'width'},  # shape(1,3,640,640)
                                        'output': {0: 'batch', 1: 'anchors'}  # shape(1,25200,85)
                                        } if dynamic else None)

        # Checks
        model_onnx = onnx.load(f)  # load onnx model
        onnx.checker.check_model(model_onnx)  # check onnx model
        # print(onnx.helper.printable_graph(model_onnx.graph))  # print

        # Simplify
        if simplify:
            try:
                import onnxsim

                print(f'{prefix} simplifying with onnx-simplifier {onnxsim.__version__}...')
                model_onnx, check = onnxsim.simplify(
                    model_onnx,
                    dynamic_input_shape=dynamic,
                    input_shapes={'images': list(img.shape)} if dynamic else None)
                assert check, 'assert check failed'
                onnx.save(model_onnx, f)
            except Exception as e:
                print(f'{prefix} simplifier failure: {e}')
        print(f'{prefix} export success, saved as {f} ({file_size(f):.1f} MB)')
        print(f"{prefix} run --dynamic ONNX model inference with: 'python detect.py --weights {f}'")
    except Exception as e:
        print(f'{prefix} export failure: {e}')


def export_coreml(model, img, file):
    # CoreML model export
    prefix = colorstr('CoreML:')
    try:
        check_requirements(('coremltools',))
        import coremltools as ct

        print(f'\n{prefix} starting export with coremltools {ct.__version__}...')
        f = file.with_suffix('.mlmodel')
        model.train()  # CoreML exports should be placed in model.train() mode
        ts = torch.jit.trace(model, img, strict=False)  # TorchScript model
        model = ct.convert(ts, inputs=[ct.ImageType('image', shape=img.shape, scale=1 / 255.0, bias=[0, 0, 0])])
        model.save(f)
        print(f'{prefix} export success, saved as {f} ({file_size(f):.1f} MB)')
    except Exception as e:
        print(f'\n{prefix} export failure: {e}')


def run(weights='model.pt',  # weights path
        img_size=(640, 640),  # image (height, width)
        batch_size=1,  # batch size
        device='cpu',  # cuda device, i.e. 0 or 0,1,2,3 or cpu
        include=('torchscript', 'onnx', 'coreml'),  # include formats
        half=False,  # FP16 half-precision export
        inplace=True,  # set YOLOv5 Detect() inplace=True
        train=False,  # model.train() mode
        optimize=False,  # TorchScript: optimize for mobile
        dynamic=False,  # ONNX: dynamic axes
        simplify=False,  # ONNX: simplify model
        opset=10,  # ONNX: opset version
        ):
    t = time.time()
    include = [x.lower() for x in include]
    img_size *= 2 if len(img_size) == 1 else 1  # expand
    file = Path(weights)

    # Load PyTorch model
    device = select_device(device)
    assert not (device.type == 'cpu' and half), '--half only compatible with GPU export, i.e. use --device 0'
    model = attempt_load(weights, map_location=device)  # load FP32 model
    names = model.names

    # Input
    gs = int(max(model.stride))  # grid size (max stride)
    img_size = [check_img_size(x, gs) for x in img_size]  # verify img_size are gs-multiples
    img = torch.zeros(batch_size, 3, *img_size).to(device)  # image size(1,3,320,192) iDetection

    # Update model
    if half:
        img, model = img.half(), model.half()  # to FP16
    model.train() if train else model.eval()  # training mode = no Detect() layer grid construction
    for k, m in model.named_modules():
        if isinstance(m, Conv):  # assign export-friendly activations
            if isinstance(m.act, nn.Hardswish):
                m.act = Hardswish()
            elif isinstance(m.act, nn.SiLU):
                m.act = SiLU()
        elif isinstance(m, Detect):
            m.inplace = inplace
            m.onnx_dynamic = dynamic
            # m.forward = m.forward_export  # assign forward (optional)

    for _ in range(2):
        y = model(img)  # dry runs
    print(f"\n{colorstr('PyTorch:')} starting from {weights} ({file_size(weights):.1f} MB)")

    # Exports
    if 'torchscript' in include:
        export_torchscript(model, img, file, optimize)
    if 'onnx' in include:
        export_onnx(model, img, file, opset, train, dynamic, simplify)
    if 'coreml' in include:
        export_coreml(model, img, file)

    # Finish
    print(f'\nExport complete ({time.time() - t:.2f}s)'
          f"\nResults saved to {colorstr('bold', file.parent.resolve())}"
          f'\nVisualize with https://netron.app')


def parse_opt():
    parser = argparse.ArgumentParser()
    parser.add_argument('--weights', type=str, default='model.pt', help='weights path')
    parser.add_argument('--img-size', nargs='+', type=int, default=[640, 640], help='image (height, width)')
    parser.add_argument('--batch-size', type=int, default=1, help='batch size')
    parser.add_argument('--device', default='cpu', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
    parser.add_argument('--include', nargs='+', default=['torchscript', 'onnx', 'coreml'], help='include formats')
    parser.add_argument('--half', action='store_true', help='FP16 half-precision export')
    parser.add_argument('--inplace', action='store_true', help='set YOLOv5 Detect() inplace=True')
    parser.add_argument('--train', action='store_true', help='model.train() mode')
    parser.add_argument('--optimize', action='store_true', help='TorchScript: optimize for mobile')
    parser.add_argument('--dynamic', action='store_true', help='ONNX: dynamic axes')
    parser.add_argument('--simplify', action='store_true', help='ONNX: simplify model')
    parser.add_argument('--opset', type=int, default=12, help='ONNX: opset version')
    opt = parser.parse_args()
    return opt


def main(opt):
    set_logging()
    print(colorstr('export: ') + ', '.join(f'{k}={v}' for k, v in vars(opt).items()))
    run(**vars(opt))


if __name__ == "__main__":
    opt = parse_opt()
    main(opt)
`

4. Run openvino environment : $cd Program Files/Intel/openvino/bin/ the run $setupvars.bat
5. Convert ONNX file to IR format : $cd Program Files/Intel/openvino/deployment tools/model optimizer
   $ python mo onnx.py –input model model.onnx –model name output -s 255
   –reverse input channels –output Conv 339,Conv 291,Conv 243 (please use Netron to check layers names)

if it didn't work, don't hesitate to send me an email: bessmagsm@gmail.com or ask here.

[aseprohman]

thanks @besmaguesmi, I will try it.

have you ever tried to compare performance of model inference between YOLOv5 in IR model while executed by the openvino framework and the yolov5 executed by the pytorch framework ?
Openvino have some optimization tools for speed up model performance like Post-training Optimization Tool (POT) and Neural Network Compression Framework (NNCF). have you ever tried it ?

[besmaguesmi]

HI @aseprohman, yes, of course, I compared the inference time as well as Throughput of the model (FPS), you can use DL benchmark to convert the model from FP16 to INT8 but unfortunately, you couldn't use the INT8 model with VPU device, otherwise, I highly recommend to use MYRIADX which provide me 10 times much better result than CPU: https://www.intel.com/content/www/us/en/products/details/processors/movidius-vpu/movidius-myriad-x.html

Good luck.

[aseprohman]

thanks a lot @besmaguesmi for your explanation
maybe I'll ask you again next time after doing an experiment :)
regards

[aseprohman]

this message appears when I try to export pt model to onnx model

TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if self.grid[i].shape[2:4] != x[i].shape[2:4] or self.onnx_dynamic:

I do export command like this

python3 export.py --weights weights/yolov5m.pt --include onnx --device cpu --batch-size 1 --img 1024

I do training with --img 416 but I want to run inference with 1024 cause I have better accuracy result when run inference in bigger size. am I wrong ?

[aseprohman]

@glenn-jocher or @besmaguesmi, do you have any suggestion ?

[glenn-jocher]

@aseprohman you can ignore warnings

[besmaguesmi]

Hi @aseprohman, sorry for the later reply! first I don't agree with you to increase the image size in inference! why did you use Openvino? to speed up the inference right? so when the image size increases the inference time increase (read this: https://www.researchgate.net/figure/The-impact-of-image-size-on-the-inference-speed-on-an-edge-device_fig9_323867606) In addition to obtaining a good bounding box size I highly recommend to use the same image size of the model (for Yolov5m model use image size = 640) for training. here is what exactly you have to do after training the model with 640 image size (Yolov5m):
$python export.py –weights model.pt –img 640 –batch 1
Looking forward to your result. Good luck

[besmaguesmi]

Another modification maybe you have to do in export.py
model.model[-1].export = True
When set to True, Detect layers (including nms, anchor frame calculations, etc.) are not output to the
model.
Models set to False with Detect layer cannot be converted from onnx to OpenVINO format models.

[zzff-sys]

Hello! I use the command line to convert ONNX files to IR format: python mo_onnx.py --input_model=yolov5s.onnx --model_name=test -s 255 --reverse_input_channels --output Conv_416,Conv_482,Conv_350 --data_type=FP16 (Layer name already viewed with Netron)，But the following error is returned：[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.front.user_data_repack.UserDataRepack'>): No node with name Conv_416

How can I solve this problem, thank you very @besmaguesmi @glenn-jocher