[Web] Deformable-DETR inference takes 12 seconds on WebGPU, 0.05s in Python (CUDAProvider)

[Dexterp37]

Describe the issue

I'm trying to run inference on a Deformable-DETR model trained using HF Transformers and converted to onnx. Running inference using onnxruntime-gpu in Python works like a charm, with the expected performance (0.05s). Running the same model in onnxruntime-web using the WebGPU provider takes a bit less than using WASM, about 12 seconds (after warmup).

I tried to follow the docs to collect some additional information and I get quite a few instances like these:

ort-wasm-simd-threaded.jsep.wasm:0x1039a7f 2024-10-13 21:42:56.842600 [V:onnxruntime:Default, js_execution_provider.cc:735 JsExecutionProvider] Graph capture enable: 0
ort-wasm-simd-threaded.jsep.wasm:0x1039a7f 2024-10-13 21:42:57.538899 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /model/model/input_proj.3/input_proj.3.1/Reshape_1

Unfortunately the analyzing the profiling data section is under construction, so I'm not sure how to act on the above information. Any help appreciated!

To reproduce

The converted model file is available here.

The code to reproduce the problem is available as a gist here: simply load the model and it will attempt to run inference twice and measure the time.

Urgency

The world is not going to end, but my research project is blocked on this 😢

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.20.0-dev.20241012-332173509d

Execution Provider

'webgpu' (WebGPU)

[gyagp]

Could you please share the model and sample code, so that we can understand what happenes to WebGPU EP? Thanks!

[Dexterp37]

Could you please share the model and sample code, so that we can understand what happenes to WebGPU EP? Thanks!

@gyagp thank you so much for your quick reply. I amended my initial report with links to the model and the code.

[gyagp]

Thanks for the model and sample code! We did some analysis, and the major reasons of slowness are related to unimplemented op (GridSample) and int64 data type, so that we have too many data copies between main memory and gpu memory.
We will improve this in the following days.

[GridSample is not implemented in WebGPU EP]

[Memory copies]

[Dexterp37]

Thanks for the model and sample code! We did some analysis, and the major reasons of slowness are related to unimplemented op (GridSample) and int64 data type, so that we have too many data copies between main memory and gpu memory. We will improve this in the following days.

Thank you so much for the quick turnaround on this and for your analysis efforts, @gyagp !

If it's of any help, I'm happy to contribute, although I'm not familiar with the code base and might end up slow you down :-E Anyway, thanks!

[gyagp]

Contributions are always welcome! If you have background on GPU shaders, maybe you can help the implementation of op GridSample. Int64 related improvements are either trivial or required more internal discussion (a bit challenging to support due to current framework).

[Dexterp37]

Contributions are always welcome! If you have background on GPU shaders, maybe you can help the implementation of op GridSample. Int64 related improvements are either trivial or required more internal discussion (a bit challenging to support due to current framework).

Ouch, I'm afraid I don't have a background in GPU shaders, sorry I can't be of more help !

[fs-eire]

WebGPU kernels to be implemented -

• GridSample
• ScatterND

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[Dexterp37]

Thank you all for the great work on this! The model is now behaving as expected, it's even faster than the Python version!