Optimize Transpose around QLinearSoftmax #22849
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Improved transpose around QLinearSoftmax in Level 3 NHWC Transformer. - Removed redundant code HandleQLinearConcat, HandleQLinearBinaryOp.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
yihonglyu
changed the title
skottmckay
approved these changes
Nov 18, 2024
yf711
removed
triage:approved
mszhanyi
pushed a commit
that referenced
this pull request
Nov 22, 2024
### Description <!-- Describe your changes. --> - Improved Transpose around QLinearSoftmax in Level 3 NHWC Transformer. - Removed redundant code HandleQLinearConcat, HandleQLinearBinaryOp. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> By merging and eliminating redundant transpose , the Image Segmentation i8 model (MobileNetv2 + DeepLabv3) achieves a 2.34X speedup.
guschmue
pushed a commit
that referenced
this pull request
Dec 2, 2024
### Description <!-- Describe your changes. --> - Improved Transpose around QLinearSoftmax in Level 3 NHWC Transformer. - Removed redundant code HandleQLinearConcat, HandleQLinearBinaryOp. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> By merging and eliminating redundant transpose , the Image Segmentation i8 model (MobileNetv2 + DeepLabv3) achieves a 2.34X speedup.
ankitm3k
pushed a commit
to intel/onnxruntime
that referenced
this pull request
Dec 11, 2024
### Description <!-- Describe your changes. --> - Improved Transpose around QLinearSoftmax in Level 3 NHWC Transformer. - Removed redundant code HandleQLinearConcat, HandleQLinearBinaryOp. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> By merging and eliminating redundant transpose , the Image Segmentation i8 model (MobileNetv2 + DeepLabv3) achieves a 2.34X speedup.
ankitm3k
pushed a commit
to intel/onnxruntime
that referenced
this pull request
Dec 11, 2024
### Description <!-- Describe your changes. --> - Improved Transpose around QLinearSoftmax in Level 3 NHWC Transformer. - Removed redundant code HandleQLinearConcat, HandleQLinearBinaryOp. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> By merging and eliminating redundant transpose , the Image Segmentation i8 model (MobileNetv2 + DeepLabv3) achieves a 2.34X speedup.
alex-spacemit
pushed a commit
to spacemit-com/onnxruntime
that referenced
this pull request
Jun 22, 2025
[ARM] MatMulNBits Fp16 support - API change only (microsoft#22826) A break-down PR of microsoft#22651 Op API change only. - add template to functions and classes that support fp32 and fp16 - rename functions, classes and files that support fp32 and fp16 from SQNBxxx to QNBxxx <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Change-Id: Ib489e7858d42abcbe0514ac44e4d2172e32384a3 Re-enable test symbolic shape infer (microsoft#22737) <!-- Describe your changes. --> It seems after CI updated to py310, numpy got updated to 2.0 and sympy 1.2 failed to cast float numpy array. Pointing sympy to 1.13 when py>=3.9 and re-enable unit test <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Error: Linux CPU CI [Quant tool] Handle input models with pre-quantized weights (microsoft#22633) Allows the QDQ quantizer to handle input models that already have some pre-quantized weights. In this case, the qdq quantizer will properly skip/handle the pre-quantized weights. Also handles an operator (e.g., Conv) with a pre-quantized weight and a float bias. The tool will read the pre-quantized weight's quantization scale to compute the bias's scale (`bias_scale = input_scale * weight_scale`). Input model (pre-quantized Conv weight):  Output QDQ model (everything is quantized):  Customers may use external tools to quantize some weights (e.g., int4 for Conv/MatMul). The qdq quantizer should still be able to quantize the rest of the model (float weights and activations) in this case. Update Gradle version 8.7 and java version 17 within onnxruntime/java (microsoft#22771) This change is to update the Gradle version within java project to 8.7, it also upgrades the JAVA to 17. Gradle version from react-native was also updated to 7.5 to make it compatible with changes from the Java directory. However, the target java version remains the same. Java version from these will be upgraded in a separated PR. This is spited from microsoft#22206 This is the first step to upgrade the react native version. Ovep develop 1.21 (microsoft#22824) OVEP development changes for ORT 1.21 Release Has critical bug fixes Support for concurrency execution of models is enabled Support for OV 2024.5 Memory optimizations for NPU platform --------- Co-authored-by: jatinwadhwa921 <[email protected]> Co-authored-by: Ankit Maheshkar <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: saurabhkale17 <[email protected]> Co-authored-by: TejalKhade28 <[email protected]> Co-authored-by: Javier E. Martinez <[email protected]> Fix 1.20 cuda minimal build failure (microsoft#22751) Fixes build failure for the cuda minimal build <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> [This change](microsoft#19470) in 1.20 is causing build failures for the cuda minimal build. Essentially, some cudnn logic was not guarded by the `USE_CUDA_MINIMAL`. Also the build is looking for cudnn while in the cuda minimal build it shouldn't depend on it, resulting in linking error. cc @gedoensmax @chilo-ms [ARM] MatMulNBits fp16 support - connect kernels (microsoft#22856) A breakdown PR of microsoft#22651 <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Change-Id: I3014c1002ff375a507bc04de7756baacf9a2b77a [WebNN EP] Support Einsum op (microsoft#19558) Adds support for einsum via WebNN matmul, transpose, reshape, reducesum, identity and element-wise binary ops. Refactor SkipLayerNorm and handle beta properly (microsoft#22862) Signed-off-by: Liqun Fu <[email protected]> Signed-off-by: Liqun Fu <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Change-Id: Ic5b8a6eb775542a57f07f5e593cc399dd7eeaa8f Fix CUDA/DML package exception caused by ENABLE_CUDA_NHWC_OPS (microsoft#22851) Now, ENABLE_CUDA_NHWC_OPS is enabled by default. It adds a new chance to create cuda provider while both cuda/dml are enabled <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Optimize Transpose around QLinearSoftmax (microsoft#22849) <!-- Describe your changes. --> - Improved Transpose around QLinearSoftmax in Level 3 NHWC Transformer. - Removed redundant code HandleQLinearConcat, HandleQLinearBinaryOp. <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> By merging and eliminating redundant transpose , the Image Segmentation i8 model (MobileNetv2 + DeepLabv3) achieves a 2.34X speedup. Replace INFINITY by std::numeric_limits<float>::infinity() (microsoft#22868) Replace INFINITY by `std::numeric_limits<float>::infinity()` to avoid build errors with Visual Studio 2022 v17.12 Preview 5 microsoft#22728 [js/webgpu] Optimize transpose as reshape when suitable (microsoft#22870) BUG microsoft#22031 Change-Id: I6c70d84228f1563792218c6c3c18b023852d4147 clang format code Change-Id: I422a9474da9e9cfc9ac8819569a13520c5d2641f
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Motivation and Context
By merging and eliminating redundant transpose , the Image Segmentation i8 model (MobileNetv2 + DeepLabv3) achieves a 2.34X speedup.