option for edge devices

[tino926]

Search before asking

• I have searched the YOLOv5 issues and found no similar feature requests.

Description

An option to train with relu activation.
An option to export model without final permutation and translation layers

Use case

With these modifications, the model is easier to be ported to edge devices and with better performance, such as google's edgetpu and rockchip's npu

Additional

YOLOv5 is the most easy tool I have tried to train a detection model (I've tried yolov4, mt-yolo, yolov6, ssd-mobiledet, yolov5-lite, ...).

However, for efficient inferring on edge device (google edgetpu, rockchip rk-xxxx, intel movidius, ...), we need to modify the activation function (to relu) and remove some layers. With these modification, we've got good results, for example we can run a 6-classes yolov5n model on rk-3568 with 40 fps and on google edgetpu with 200 fps (including c++ implemented post processing such as nms).

It's not hard for user to modify the source code to achieve these. However, I feel it worth being added to the official codes.

This is an example to run our model on google's edgetpu.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@tino926 thanks for the feedback! We do have simpler activations in our roadmap. The plan is to add an activation: field to the yaml, with default/undefined setting to nn.SiLU, and manual setting to any other activation, i.e.

activation: nn.LeakyReLU(0.1)

It may be a while before we can complete this, but if you'd like to help contribute this, we are open to PRs!

Please see our ✅ Contributing Guide to get started.

[glenn-jocher]

@tino926 good news 😃! Your original issue may now be partially improved ✅ in PR #9371. This PR adds support for other activation functions in model yamls.

To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[Lebhoryi]

Thx for leaky relu supported with tagv7.
I have try to change default activation silu to leakyrelu by yolov5s6.yaml, but all of the boxes are wrong.

Q：Where can I get the yolov5s6 with leaky relu?

[glenn-jocher]

@Lebhoryi you can't change activations after a model has been trained. This is for training new models.

[Tsardoz]

@tino926 Which layers did you remove for optimal performance please?

[tino926]

@Tsardoz,
These layers should not be quantized or should be run on the CPU. On the other hand, you may want to try YOLOv6-4.0, as it now has an option to export models that are friendly to edge devices.

[glenn-jocher]

@tino926, certain layers may not be suitable for quantization or optimal for some edge devices when run on the GPU. Regarding your question, we suggest checking out YOLOv6-4.0 with its optimized model exports that are friendly to edge devices.

Thank you for your interest in our project and please let us know if you have any further questions.

[tino926]

@glenn-jocher
I really hope there is an official option in yolov5 to generate models without those layers, as it is currently the fastest yolo-related model I have tried for running in an openvino/int8/cpu environment. Currently, I modify the Detect class in yolo.py whenever I need to export models. While I am willing to make a contribution, I am not confident in my ability to complete this task well (I don't know how to modify export.py in a good way)

[glenn-jocher]

@tino926 thank you for your feedback and for sharing your experience with running YOLOv5 on openvino/int8/cpu. We appreciate your interest in contributing to the project!

We are currently working on optimizing model exports for edge devices, and we hope to include an option to generate models without certain layers in the near future. In the meantime, please feel free to contribute to the project in any way that you feel comfortable with, even if it's just reporting issues or suggesting new features.

If you have any specific questions or concerns about modifying the export.py file, please don't hesitate to ask. We are happy to help in any way we can!

[Tsardoz]

@tino926 Thanks for that. Could you point me to how to split a model and run some of it on CPU and the rest on NPU? I was not aware this could even be done. I shall look forward to a YOLOv5/8 release which is edge friendly.

[glenn-jocher]

@Tsardoz Splitting a model between CPU and NPU can be done via model parallelism, where different portions of the model are run on different devices. The specifics of how to do this depend on the specific hardware and software stack you are using, so I recommend consulting the documentation for your platform or reaching out to the manufacturer's support team for assistance.

Regarding a YOLOv5/8 release that is more edge-friendly, we are constantly working on improving our models to run efficiently on different environments and hardware. Keep an eye on future releases and updates to our repository for new features and optimizations.

Thank you for your interest in our project and please let us know if you have any further questions or concerns.

[tino926]

@Tsardoz
This is what I have done:

1. Obtain the .onnx model without those layers (you can modify the Detect class in yolo.py or directly use onnx-modifier to do this.
2. quantize and convert the .onnx model to .rknn and using rockchip's tool
3. implement c++ code to interpret the output from the the .rknn model referencing the Detect class in yolo.py

My code are located in several files and classes. It would be meaningless if I just extract them to here without detailed explanation. So ... you need to do some work by yourself.

Please note that I have chosen this approach only because it is suitable for various devices. You may find better way for your working scenario. In my experience, when I exported the model to tflite (before YOLOv5 officially supported it), the screen shows that some layers are not supported and will be run on CPU. Thus I guess for certain devices, it may be possible to run portion of the model on CPU, but I have not conducted any research on this matter.

[Tsardoz]

@tino926
Thanks that is a huge help. So it sounds like you have just deleted these layers rather than doing model partitioning (splitting the layers between CPU and NPU). I have found some academic papers on doing this but have not yet looked into it. It sounds next level for me.