Why is the TensorRT input binding set to FP16 for FP16 engines by default?

[DavidBaldsiefen]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

No response

Bug

Right now, when exporting a model to TensorRT using export.py with the --half argument, the input binding is set to datatype "half" while the output is set to datatype "float":

TensorRT: Network Description:
TensorRT:	input "images" with shape (1, 3, 640, 640) and dtype DataType.HALF
TensorRT:	output "output" with shape (1, 25200, 6) and dtype DataType.FLOAT

However, this means that whenever the resulting engine is used, its inputs also need to be converted to FP16, as can be seen in detect.py:

yolov5/detect.py

Line 118 in cea994b

im = im.half() if half else im.float() # uint8 to fp16/32

Now I was wondering: is this intentional? TensorRT can handle an input in FP32 without problem even if the remaining engine is in half precision. However, having TensorRT handle the conversion means less overhead for the developer, while most likely also being faster as the conversion can be handled by the GPU instead of CPU. The datatype of the input binding when creating a new engine can be easily changed with a single line of code:

network.get_input(0).dtype = trt.DataType.FLOAT

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@DavidBaldsiefen thanks for raising this issue!

The basic idea is that --half export will create a TRT model expecting --half inputs, i.e.

python export.py --half
python detect.py --half

If you want FP32 inputs simply use default settings:

python export.py
python detect.py

Does this resolve your question or do you still think there's an optimization/simplification we could do in the workflow?

[DavidBaldsiefen]

I believe that it makes more sense to set the default input binding to float even for FP16 engines. As mentioned above, I do belive that a) it is faster to have TensorRT handle the input conversion and b) it is more intuitive for developers. For example, C++ does not have any FP16 datatype by default, which means that the engine exported by export.py cannot be used for C++ inference without modifications. In addition, it would no longer be necessary to append the --half option to val.py or detect.py. However, I am not sure how changing the input-tensor to float precision would affect ONNX models that are converted to FP16

[glenn-jocher]

@DavidBaldsiefen hmm interesting. I just tested a new workflow without issue, exporting at --half and running inference with FP32 inputs. The TRT model seems to convert on it's own (so flag already appears unnecessary).

python export.py --half
python detect.py

[glenn-jocher]

@DavidBaldsiefen BTW the ONNX exports made during TRT export are just a byproduct of the TRT exports, so I wouldn't worry about them, they likely won't be used by users. You can always export them directly with --include onnx also.

[DavidBaldsiefen]

I just tested a new workflow without issue, exporting at --half and running inference with FP32 inputs. The TRT model seems to convert on it's own (so flag already appears unnecessary).

How so? When I create an engine using the default export.py, I need to set the --half flag for detect.py and val.py, as otherwise nothing is detected. If I modify export.py so that the input binding is FP32, the opposite is true.

[glenn-jocher]

@DavidBaldsiefen ah yes, you are correct! Silent errors (no detections) are produced with improper FP16-FP32 pairing, but no error is shown unfortunately.

[glenn-jocher]

I get these results when testing all 4 combinations:

[DavidBaldsiefen]

The results make sense. A start could be to check the engines input datatype in detect.py (or common.py) and print a warning if it does not match the presence (or absence) of --half. That would prevent issues like #6766

[glenn-jocher]

@DavidBaldsiefen do you know of a way to determine the quantization of a TRT model after loading? Perhaps we should have an assert but I don't know how to get the actual value, i.e.:

bits = get_TRT_quantization(trt_model)
assert bits == (16 if half else 32), 'model ' + ('requires' if bits == 16 else 'incompatible with') + ' --half'

[DavidBaldsiefen]

@glenn-jocher Yes, I have created a pull request that would do the assertion in val.py and detect.py : #6798

[glenn-jocher]

@DavidBaldsiefen I was thinking after seeing the above issues perhaps we want to default TRT exports to FP16 and then add the line in your original post since you said this could accommodate both FP32 and FP16 inputs?

network.get_input(0).dtype = trt.DataType.FLOAT

Part of what we want to do is default to the best settings, and based on the above speed results --half TRT export seems like the best choice. I still need to do a mAP comparison, but usually (i.e. with PyTorch) FP32 and FP16 inference return almost identical accuracies.

EDIT: where exactly would we slot the above line into export.py?

[glenn-jocher]

@DavidBaldsiefen any comment on what I asked earlier in #6777 (comment) ?

Would it make sense to default to FP16 and add your line to be robust to FP32 inputs by default?