Ovep develop 1.21 #22824
Merged
Ovep develop 1.21 #22824
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add check for NPU device type in allocator creation logic
add initializer data type support
Create multi infer requests in OVEP based on num_threads
update: Upgraded to OpenVINO v2024.5
update: Extend datatypes support
Updated API Documentation with required comments
* Add support for dynmaic workload type * Fix the iteration on gsl::span dynamic keys and values * Fix lint issues
Fix inference with EP context embed mode 0
Upgrade OVEP CI to 2024.4.0
|
/azp run Linux OpenVINO CI Pipeline |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@adrianlizarraga @jywu-msft Please review |
|
please fix lintrunner error https://github.com/microsoft/onnxruntime/actions/runs/11822465741/job/32940502475?pr=22824 |
|
@jywu-msft Added the fix for pylint failure. Can you please retrigger the pipelines. Thanks |
|
/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, Linux QNN CI Pipeline |
|
/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows ARM64 QNN CI Pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, Windows x64 QNN CI Pipeline, Linux MIGraphX CI Pipeline, Big Models |
|
/azp run Linux Android Emulator QNN CI Pipeline, Windows GPU CUDA CI Pipeline, Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 6 pipeline(s). |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
Azure Pipelines successfully started running 9 pipeline(s). |
jywu-msft
approved these changes
Nov 15, 2024
guschmue
pushed a commit
that referenced
this pull request
Dec 2, 2024
### Description OVEP development changes for ORT 1.21 Release ### Motivation and Context Has critical bug fixes Support for concurrency execution of models is enabled Support for OV 2024.5 Memory optimizations for NPU platform --------- Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: Ankit Maheshkar <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: saurabhkale17 <[email protected]> Co-authored-by: TejalKhade28 <[email protected]> Co-authored-by: Javier E. Martinez <[email protected]>
ankitm3k
added a commit
to intel/onnxruntime
that referenced
this pull request
Dec 11, 2024
OVEP development changes for ORT 1.21 Release Has critical bug fixes Support for concurrency execution of models is enabled Support for OV 2024.5 Memory optimizations for NPU platform --------- Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: Ankit Maheshkar <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: saurabhkale17 <[email protected]> Co-authored-by: TejalKhade28 <[email protected]> Co-authored-by: Javier E. Martinez <[email protected]>
ankitm3k
added a commit
to intel/onnxruntime
that referenced
this pull request
Dec 11, 2024
OVEP development changes for ORT 1.21 Release Has critical bug fixes Support for concurrency execution of models is enabled Support for OV 2024.5 Memory optimizations for NPU platform --------- Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: Ankit Maheshkar <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: saurabhkale17 <[email protected]> Co-authored-by: TejalKhade28 <[email protected]> Co-authored-by: Javier E. Martinez <[email protected]>
ankitm3k
added a commit
to intel/onnxruntime
that referenced
this pull request
Dec 11, 2024
OVEP development changes for ORT 1.21 Release Has critical bug fixes Support for concurrency execution of models is enabled Support for OV 2024.5 Memory optimizations for NPU platform --------- Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: Ankit Maheshkar <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: saurabhkale17 <[email protected]> Co-authored-by: TejalKhade28 <[email protected]> Co-authored-by: Javier E. Martinez <[email protected]>
alex-spacemit
pushed a commit
to spacemit-com/onnxruntime
that referenced
this pull request
Jun 22, 2025
[ARM] MatMulNBits Fp16 support - API change only (microsoft#22826) A break-down PR of microsoft#22651 Op API change only. - add template to functions and classes that support fp32 and fp16 - rename functions, classes and files that support fp32 and fp16 from SQNBxxx to QNBxxx <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Change-Id: Ib489e7858d42abcbe0514ac44e4d2172e32384a3 Re-enable test symbolic shape infer (microsoft#22737) <!-- Describe your changes. --> It seems after CI updated to py310, numpy got updated to 2.0 and sympy 1.2 failed to cast float numpy array. Pointing sympy to 1.13 when py>=3.9 and re-enable unit test <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Error: Linux CPU CI [Quant tool] Handle input models with pre-quantized weights (microsoft#22633) Allows the QDQ quantizer to handle input models that already have some pre-quantized weights. In this case, the qdq quantizer will properly skip/handle the pre-quantized weights. Also handles an operator (e.g., Conv) with a pre-quantized weight and a float bias. The tool will read the pre-quantized weight's quantization scale to compute the bias's scale (`bias_scale = input_scale * weight_scale`). Input model (pre-quantized Conv weight):  Output QDQ model (everything is quantized):  Customers may use external tools to quantize some weights (e.g., int4 for Conv/MatMul). The qdq quantizer should still be able to quantize the rest of the model (float weights and activations) in this case. Update Gradle version 8.7 and java version 17 within onnxruntime/java (microsoft#22771) This change is to update the Gradle version within java project to 8.7, it also upgrades the JAVA to 17. Gradle version from react-native was also updated to 7.5 to make it compatible with changes from the Java directory. However, the target java version remains the same. Java version from these will be upgraded in a separated PR. This is spited from microsoft#22206 This is the first step to upgrade the react native version. Ovep develop 1.21 (microsoft#22824) OVEP development changes for ORT 1.21 Release Has critical bug fixes Support for concurrency execution of models is enabled Support for OV 2024.5 Memory optimizations for NPU platform --------- Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: Ankit Maheshkar <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: saurabhkale17 <[email protected]> Co-authored-by: TejalKhade28 <[email protected]> Co-authored-by: Javier E. Martinez <[email protected]> Fix 1.20 cuda minimal build failure (microsoft#22751) Fixes build failure for the cuda minimal build <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> [This change](microsoft#19470) in 1.20 is causing build failures for the cuda minimal build. Essentially, some cudnn logic was not guarded by the `USE_CUDA_MINIMAL`. Also the build is looking for cudnn while in the cuda minimal build it shouldn't depend on it, resulting in linking error. cc @gedoensmax @chilo-ms [ARM] MatMulNBits fp16 support - connect kernels (microsoft#22856) A breakdown PR of microsoft#22651 <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Change-Id: I3014c1002ff375a507bc04de7756baacf9a2b77a [WebNN EP] Support Einsum op (microsoft#19558) Adds support for einsum via WebNN matmul, transpose, reshape, reducesum, identity and element-wise binary ops. Refactor SkipLayerNorm and handle beta properly (microsoft#22862) Signed-off-by: Liqun Fu <[email protected]> Signed-off-by: Liqun Fu <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Change-Id: Ic5b8a6eb775542a57f07f5e593cc399dd7eeaa8f Fix CUDA/DML package exception caused by ENABLE_CUDA_NHWC_OPS (microsoft#22851) Now, ENABLE_CUDA_NHWC_OPS is enabled by default. It adds a new chance to create cuda provider while both cuda/dml are enabled <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Optimize Transpose around QLinearSoftmax (microsoft#22849) <!-- Describe your changes. --> - Improved Transpose around QLinearSoftmax in Level 3 NHWC Transformer. - Removed redundant code HandleQLinearConcat, HandleQLinearBinaryOp. <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> By merging and eliminating redundant transpose , the Image Segmentation i8 model (MobileNetv2 + DeepLabv3) achieves a 2.34X speedup. Replace INFINITY by std::numeric_limits<float>::infinity() (microsoft#22868) Replace INFINITY by `std::numeric_limits<float>::infinity()` to avoid build errors with Visual Studio 2022 v17.12 Preview 5 microsoft#22728 [js/webgpu] Optimize transpose as reshape when suitable (microsoft#22870) BUG microsoft#22031 Change-Id: I6c70d84228f1563792218c6c3c18b023852d4147 clang format code Change-Id: I422a9474da9e9cfc9ac8819569a13520c5d2641f
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
OVEP development changes for ORT 1.21 Release
Motivation and Context
Has critical bug fixes
Support for concurrency execution of models is enabled
Support for OV 2024.5
Memory optimizations for NPU platform