TRT errors on concatentation route_15

[deanofthewebb]

Total weights read: 37669889
Building YOLO network

      layer                        input               output         weightPtr
(0)   conv_silu                  3 x 640 x 640      32 x 640 x 640    992    
(1)   conv_silu                 32 x 640 x 640      64 x 320 x 320    19680  
(2)   conv_silu                 64 x 320 x 320      64 x 320 x 320    56800  
(3)   conv_silu                 64 x 320 x 320     128 x 160 x 160    131040 
(4)   conv_silu                128 x 160 x 160      64 x 160 x 160    139488 
(5)   conv_silu                 64 x 160 x 160      64 x 160 x 160    143840 
(6)   conv_silu                 64 x 160 x 160      64 x 160 x 160    180960 
(7)   conv_silu                 64 x 160 x 160      64 x 160 x 160    218080 
(8)   conv_silu                 64 x 160 x 160      64 x 160 x 160    255200 
(9)   conv_silu                 64 x 160 x 160      64 x 160 x 160    292320 
(10)  route                           -            128 x 160 x 160    292320 
(11)  conv_silu                128 x 160 x 160     256 x 160 x 160    326112 
(12)  conv_silu                256 x 160 x 160     128 x 160 x 160    359392 
(13)  conv_silu                128 x 160 x 160     128 x 160 x 160    376288 
(14)  conv_silu                128 x 160 x 160     128 x  80 x  80    524256 
ERROR: [TRT]: 4: [layers.cpp::estimateOutputDims::1944] Error Code 4: Internal Error (route_15: all concat input tensors must have the same dimensions except on the concatenation axis (0), but dimensions mismatched at index 1. Input 0 shape: [128,80,80], Input 1 shape: [128,160,160])

I tried my hand at generating darknet weights for yolov7.pt but I am blocked getting it converted to TensorRT. Any guidance on this issue?

[WongKinYiu]

.yaml could conv directly from specific layer, .cfg need route first then conv to the routed layer.
https://github.com/WongKinYiu/yolov7/blob/main/cfg/deploy/yolov7.yaml#L22

[deanofthewebb]

Ok thanks! @WongKinYiu

[philipp-schmidt]

@deanofthewebb
what are you using to convert to TensorRT? Where's that output from that you pasted?

[deanofthewebb]

@philipp-schmidt Deepstream

[philipp-schmidt]

@deanofthewebb

I tried converting with trtexec (prebuilt in the TensorRT NGC image) and I'm unable to do so if the ONNX export is with --grid enabled.

Were you successful in converting to TensorRT with the Detect() layer enabled? Or are you using a TensorRT plugin?

Issue is described here: #66

[deanofthewebb]

@philipp-schmidt This one works: #57

[philipp-schmidt]

Thanks for the pointer.
It's not using the --grid option on ONNX export though if I see that correctly.
You tested it with --grid?

[deanofthewebb]

@philipp-schmidt Yes, it indeed works with grid

[philipp-schmidt]

Same issue unfortunately. Anything you changed to get it working on your end?

Export via: python models/export.py --weights yolov7.pt --grid
Conversion:

root@3aa30b614471:/workspace/yolov7# python deploy/TensoorRT/onnx_to_tensorrt.py --onnx yolov7.onnx --fp16 --explicit-batch -o yolov7.engine
Namespace(calibration_batch_size=128, calibration_cache='calibration.cache', calibration_data=None, debug=False, explicit_batch=True, explicit_precision=False, fp16=True, gpu_fallback=False, int8=False, max_batch_size=None, max_calibration_size=2048, onnx='yolov7.onnx', output='yolov7.engine', refittable=False, simple=False, strict_types=False, verbosity=None)
2022-07-10 07:53:52 - __main__ - INFO - TRT_LOGGER Verbosity: Severity.ERROR
2022-07-10 07:53:52 - __main__ - INFO - Setting BuilderFlag.FP16
[TensorRT] ERROR: [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_378: broadcast dimensions must be conformable
)
ERROR: Failed to parse the ONNX file: yolov7.onnx
In node 378 (parseGraph): INVALID_NODE: Invalid Node - Mul_378
[graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_378: broadcast dimensions must be conformable)

[deanofthewebb]

Try it with batch size 1

…

On Sun, Jul 10, 2022 at 12:57 AM philipp-schmidt ***@***.***> wrote: Same issue unfortunately. Anything you changed to get it working on your end? Export via: python models/export.py --weights yolov7.pt --grid Conversion: ***@***.***:/workspace/yolov7# python deploy/TensoorRT/onnx_to_tensorrt.py --onnx yolov7.onnx --fp16 --explicit-batch -o yolov7.engine Namespace(calibration_batch_size=128, calibration_cache='calibration.cache', calibration_data=None, debug=False, explicit_batch=True, explicit_precision=False, fp16=True, gpu_fallback=False, int8=False, max_batch_size=None, max_calibration_size=2048, onnx='yolov7.onnx', output='yolov7.engine', refittable=False, simple=False, strict_types=False, verbosity=None) 2022-07-10 07:53:52 - __main__ - INFO - TRT_LOGGER Verbosity: Severity.ERROR 2022-07-10 07:53:52 - __main__ - INFO - Setting BuilderFlag.FP16 [TensorRT] ERROR: [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_378: broadcast dimensions must be conformable ) ERROR: Failed to parse the ONNX file: yolov7.onnx In node 378 (parseGraph): INVALID_NODE: Invalid Node - Mul_378 [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_378: broadcast dimensions must be conformable) — Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPNYZM4LHZEVQ3IAVMR46DVTJ67ZANCNFSM527HCUOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[deanofthewebb]

@philipp-schmidt

python models/export.py --weights /content/yolov7.pt --grid
usr/local/lib/python3.7/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.10) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Namespace(batch_size=1, device='cpu', dynamic=False, grid=True, img_size=[640, 640], weights='/content/yolov7.pt')
YOLOR 🚀 v0.1-8-gdfc6817 torch 1.11.0+cu113 CPU

Fusing layers... 
RepConv.fuse_repvgg_block
RepConv.fuse_repvgg_block
RepConv.fuse_repvgg_block
Model Summary: 306 layers, 36905341 parameters, 36905341 gradients
/usr/local/lib/python3.7/dist-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2228.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]

Starting TorchScript export with torch 1.11.0+cu113...
./models/yolo.py:49: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.grid[i].shape[2:4] != x[i].shape[2:4]:
TorchScript export success, saved as /content/yolov7.torchscript.pt

Starting ONNX export with onnx 1.12.0...
ONNX export success, saved as /content/yolov7.onnx
CoreML export failure: No module named 'coremltools'

Export complete (11.78s). Visualize with https://github.com/lutzroeder/netron.

python3 onnx_to_tensorrt.py --onnx yolov7.onnx  -b 1 -vv --explicit-batch -o  yolov7.onnx_b1_gpu0_fp16.engine --fp16 --explicit-precision

...

[07/08/2022-23:53:12] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +70, GPU +73, now: CPU 70, GPU 73 (MiB)
2022-07-08 23:53:12 - __main__ - INFO - Serializing engine to file: yolov7.onnx_b1_gpu0_fp16.engine
[07/08/2022-23:53:12] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[07/08/2022-23:53:12] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.

[philipp-schmidt]

@deanofthewebb Super weird, doesn't work for me.

root@3aa30b614471:/workspace/yolov7# python models/export.py --weights yolov7.pt --grid
Namespace(batch_size=1, device='cpu', dynamic=False, grid=True, img_size=[640, 640], weights='yolov7.pt')
YOLOR 🚀 v0.1-11-ge342736 torch 1.10.0a0+3fd9dcf CPU

Fusing layers... 
RepConv.fuse_repvgg_block
RepConv.fuse_repvgg_block
RepConv.fuse_repvgg_block
Model Summary: 306 layers, 36905341 parameters, 36905341 gradients, 104.5 GFLOPS

Starting TorchScript export with torch 1.10.0a0+3fd9dcf...
/workspace/yolov7/./models/yolo.py:49: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.grid[i].shape[2:4] != x[i].shape[2:4]:
TorchScript export success, saved as yolov7.torchscript.pt

Starting ONNX export with onnx 1.8.204...
ONNX export success, saved as yolov7.onnx
CoreML export failure: No module named 'coremltools'

Export complete (5.96s). Visualize with https://github.com/lutzroeder/netron.

root@3aa30b614471:/workspace/yolov7# python3 deploy/TensoorRT/onnx_to_tensorrt.py --onnx yolov7.onnx  -b 1 --explicit-batch -o  yolov7.onnx_b1_gpu0_fp16.engine --fp16 --explicit-precision
Namespace(calibration_batch_size=128, calibration_cache='calibration.cache', calibration_data=None, debug=False, explicit_batch=True, explicit_precision=True, fp16=True, gpu_fallback=False, int8=False, max_batch_size=1, max_calibration_size=2048, onnx='yolov7.onnx', output='yolov7.onnx_b1_gpu0_fp16.engine', refittable=False, simple=False, strict_types=False, verbosity=None)
2022-07-10 08:19:22 - __main__ - INFO - TRT_LOGGER Verbosity: Severity.ERROR
2022-07-10 08:19:23 - __main__ - INFO - Setting BuilderFlag.FP16
[TensorRT] ERROR: [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_378: broadcast dimensions must be conformable
)
ERROR: Failed to parse the ONNX file: yolov7.onnx
In node 378 (parseGraph): INVALID_NODE: Invalid Node - Mul_378
[graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_378: broadcast dimensions must be conformable
)

[deanofthewebb]

@philipp-schmidt

Try simplifying the onnx model first:
https://www.convertmodel.com/#input=onnx&output=onnx

[philipp-schmidt]

I tried that as well :D

Let me double check I'm not missing anything - thanks for the help :)

[deanofthewebb]

@philipp-schmidt Double check your TensorRT version:

pip install nvidia-pyindex
pip install nvidia-tensorrt

Mine serialized in version 213:

Serialized Engine Version: 213