AttributeError: module 'tensorflow._api.v2.lite' has no attribute 'load_delegate'

[glenn-jocher]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Export

Bug

@zldrobit I think recent changes to EdgeTPU inference created a bug where load_delegate import was attempted from tensorflow.lite.load_delegate instead of tensorflow.lite.experimental.load_delegate causing this error message:

AttributeError: module 'tensorflow._api.v2.lite' has no attribute 'load_delegate'

If I attempt a fix in #6536 I still see a new error message on EdgeTPU inference:

AttributeError: 'Delegate' object has no attribute '_library'

Environment

YOLOv5 🚀 v6.0-231-g9bc72a3 torch 1.10.0+cu111 CPU

Minimal Reproducible Example

In Colab Notebook
https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb

!python export.py --include edgetpu
!python detect.py --weights yolov5s-int8_edgetpu.tflite

Additional

Will submit PR with fix for original error.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@zldrobit ok I've merged #6536 which I think fixes original issue.

I think the second error is just because I didn't install Edge TPU runtime in Colab. Is it possible to run Edge TPU models without Edge TPU accelerator plugged into the machine? I'm asking because I have a separate branch that I'm working on that is creating a new benchmarks.py to benchmark all of the export formats (currently CPU-only), i.e.:
https://github.com/ultralytics/yolov5/blob/updates/benchmarks/utils/benchmarks.py

benchmarks: weights=/usr/src/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/usr/src/yolov5/data/coco128.yaml
...

Benchmarks complete (782.19s)
                   Weights  mAP@0.5:0.95  Inference time (ms)
0               yolov5s.pt      0.407554           102.777191
1      yolov5s.torchscript      0.402908           132.848348
2             yolov5s.onnx      0.402908            89.061137
3   yolov5s_openvino_model      0.402908            67.093970
4                   engine           NaN                  NaN
5                   coreml           NaN                  NaN
6      yolov5s_saved_model      0.402908           133.983964
7               yolov5s.pb      0.402908           101.405423
8      yolov5s-fp16.tflite      0.402851           502.289245
9                  edgetpu           NaN                  NaN
10                    tfjs           NaN                  NaN

[Rishav-hub]

Hello @glenn-jocher I am facing similar issue ModuleNotFoundError: No module named 'tensorflow.lite.experimental.load_delegate'. when trying to do this !python /content/yolov5/detect.py --weights /content/yolov5s-fp16.tflite.. Any fix?

Tensorflow Version -: 2.7.0
Environment -: colab

[glenn-jocher]

@Rishav-hub yes I think we need to follow the official example more closely, which does work in my tests:
https://www.tensorflow.org/api_docs/python/tf/lite/experimental/load_delegate

[Rishav-hub]

@glenn-jocher but what is causing this issue do I need to make changes in the common.py in yolov5. Is there any permanent fix you suggest or do I need to make changes in the source code?

[glenn-jocher]

@Rishav-hub we need a PR to update the repo with a fix. It's difficult for me to debug since I don't have any Edge TPU devices to test on. The requirements are:

• Prefer Edge TPU Runtime to TF if installed
• Fix load_delegate() issues
• Keep code as concise as possible

The area of the code that needs updating is:

yolov5/models/common.py

Lines 376 to 390 in cba4303

	elif tflite: # https://www.tensorflow.org/lite/guide/python#install_tensorflow_lite_for_python
	try: # prefer tflite_runtime if installed
	from tflite_runtime.interpreter import Interpreter, load_delegate
	except ImportError:
	import tensorflow.lite.experimental.load_delegate as load_delegate
	import tensorflow.lite.Interpreter as Interpreter
	if 'edgetpu' in w.lower(): # Edge TPU https://coral.ai/software/#edgetpu-runtime
	LOGGER.info(f'Loading {w} for TensorFlow Lite Edge TPU inference...')
	delegate = {'Linux': 'libedgetpu.so.1',
	'Darwin': 'libedgetpu.1.dylib',
	'Windows': 'edgetpu.dll'}[platform.system()]
	interpreter = Interpreter(model_path=w, experimental_delegates=[load_delegate(delegate)])
	else: # Lite
	LOGGER.info(f'Loading {w} for TensorFlow Lite inference...')
	interpreter = Interpreter(model_path=w) # load TFLite model

[glenn-jocher]

@Rishav-hub good news 😃! Your original issue may now be fixed ✅ in PR #6542. This PR fixes TF module loading only for Edge TPU TFLite exports, i.e. python detect.py --weights yolov5s-int8_edgetpu.tflite

To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!