Adapt python op converters covered in paddle-3-beta2 for TRT 10 #69510

anderson101866 · 2024-11-19T10:39:34Z

PR Category

Inference

PR Types

Bug fixes

Description

Added support for TRT 10.0 since
- some deprecated APIs were removed or (NOTE: we have a mapping table to help developer migrate code here)
- some interfaces are refined in TRT10

細節主要分為:
(1) ~~build TRT engine相關的改動~~(PR審核期間，Paddle內部已移除deprecated build_engine, see commit:d4e1784b) 、以及
(2) Layer相關的改動
- 其中(1)為一次性的改動engine建置過程；而 後續python op converter開發者僅需專注(2) 的部分，專注TRT 10的介面變化

(1)的部分請參考python/paddle/tensorrt/converter.py的改動，很簡單。可以參考上文deprecated API的表格
(2)的部分包括:

TRT10的IShapeLayer從此擴展為INT64的介面，以適應形狀大的Tensor。
- 已知某些paddle op的輸入shape_tensor只有int32版本，因此TRT上/下游的paddle op實作INT64之前，只得套用現有的workaround (see: python/paddle/tensorrt/converter_utils.py 統一轉換成較窄的int32，直到paddle正式支援int64 shape
TRT10對於pool layer僅支援nd的介面，移除2d介面 (同樣可參考上文的deprecated API表格)
- ~~see python/paddle/tensorrt/impls/pooling.py, add_pooling/stride/padding等介面移除~~
  然而此PR合併前，Paddle內部已於pooling.py中移除2d介面，see commit:6043c7f3
其餘改動，僅是驗證現有paddle-3.0.0-beta2有的單元測試有覆蓋的op，將支援版本限制放鬆
- trt_version="8.x" -> trt_version="trt_version_ge=8.0"

paddle-bot · 2024-11-19T10:39:39Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

CLAassistant · 2024-11-19T10:39:40Z

All committers have signed the CLA.

CLAassistant · 2024-11-19T10:39:40Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Anderson Meng seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

YuanRisheng · 2024-11-20T11:52:40Z

目前CI流水线上缺乏对TRT10版本的监控，PR先暂时这样提上来，可能先不合入，等最近协调后应该会上线带有TRT10的流水线，到时候再合入PR，谢谢

anderson101866 · 2024-11-21T02:46:46Z

@YuanRisheng
了解了解，另外只是確認一下工作流程:

因為現在TRT10的python converter flow 完全不能 運行，所以即使有TRT10的CI流程，CI保護的flow本身現況就是壞的，所有python converter單測現況都會failure。這是預期的對嗎?
另外，
PR-CI-APPROVAL當中回報:
1. You must have one RD (From00, zhangbo9674) approval for file changes in paddle/fluid/framework/new_executor.
屆時會有人幫我approve這底層fluid相關的改動，對吧?

YuanRisheng · 2024-11-21T03:46:01Z

@YuanRisheng 了解了解，另外只是確認一下工作流程:

因為現在TRT10的python converter flow 完全不能 運行，所以即使有TRT10的CI流程，CI保護的flow本身現況就是壞的，所有python converter單測現況都會failure。這是預期的對嗎?

另外，
PR-CI-APPROVAL當中回報:

You must have one RD (From00, zhangbo9674) approval for file changes in paddle/fluid/framework/new_executor.

屆時會有人幫我approve這底層fluid相關的改動，對吧?

@anderson101866 对于第一点你的理解应该是对的，这个我们再考虑一下，想一想是先让你们这边把修复trt10的单测pr提交合入，还是先把trt10的ci弄起来（convert相关单测禁止运行），你们这边提交修复代码后打开单测。对于第二点，你的理解也是对的，我们这边会找人来让CI-APPROVAL通过

anderson101866 · 2024-11-21T03:58:39Z

好的謝謝，另外附註，NV這邊因為環境關係，我們只有beta2的環境，所以除了build engine，op的部分只有打開幾個beta2單測有覆蓋的op。

實際develop分支的op數量應該更多，這個PR未包含

YuanRisheng · 2024-11-21T05:35:52Z

好的謝謝，另外附註，NV這邊因為環境關係，我們只有beta2的環境，所以除了build engine，op的部分只有打開幾個beta2單測有覆蓋的op。

實際develop分支的op數量應該更多，這個PR未包含

了解了，但是我们建议后续使用develop来验证trt10，beta2包含的op数量应该很少，大部分在develop里

anderson101866 · 2024-12-03T07:25:59Z

@YuanRisheng @vivienfanghuagood
Is the CI ready? or it'll continuously suffer conflict from newly submitted change.

YuanRisheng · 2024-12-03T09:01:48Z

@YuanRisheng @vivienfanghuagood Is the CI ready? or it'll continuously suffer conflict from newly submitted change.

@anderson101866 我们讨论了一下，可以先推进pr的合入，合入后我们这边线下测试一下，辛苦更新PR解决一下conflict

anderson101866 · 2024-12-05T06:46:33Z

已更新PR:

pool 2d deprecated API在此PR審核期間，已被同仁commit內部移除。故此PR捨棄對應的change
build_engine此TRT 8 API，已deprecated，也已被同仁commit內部移除，此PR同樣捨棄對應的change

YuanRisheng · 2024-12-05T11:19:44Z

CI-Codestyle-Check存在代码格式的问题，请本地修正后再提交一次，谢谢

This commit includes basic migration to TRT 10 API, and also enable those converter who are tested with existing unittest to TRT10 For those 2 converter which is NOT included in this commit: - python/paddle/tensorrt/impls/attribute.py - python/paddle/tensorrt/impls/common.py Need to fix these 2 behavior issues in later commit.

Although `IShapeLayer` supports shape in int64 since TRT10, some paddle native op kernel only implements their input shape tensor (if exists) in int32. Hence, there is a workaround in `trt_shape` to cast the result of TRT `IShapeLayer` back to int32 to be more compatible with other paddle op. (see python/paddle/tensorrt/converter_utils.py) Please remove the workaround when all paddle op supports their shape in int64. Also, since `IShapeLayer` return shape in int64 in TRT10, the "pd_op.shape64" will be seamlessly supported in TRT10 w/o any extra workaround.

Error detail: { (%1) = "pd_op.bilinear_interp" [id:28] (%2, %3, <<NULL VALUE>>, <<NULL VALUE>>) {__l_trt__:true,align_corners:false,align_mode:(Int32)0,data_format:"NCHW",interp_method:"bilinear",out_d:(Int32)-1,out_h:(Int32)12,out_w:(Int32)12,scale:[],stop_gradient:[true]} : (builtin.tensor<-1x3x6x10xf32>, builtin.tensor<2xi32>, <<NULL TYPE>>, <<NULL TYPE>>) -> builtin.tensor<-1x3x12x12xf32> () = "cf.yield" [id:36] (%1) {} : (builtin.tensor<-1x3x12x12xf32>) -> } [TRT] [E] ITensor::getDimensions: Error Code 4: API Usage Error ((Unnamed Layer* 6) [Concatenation]: concat input tensors 0 and 2 have incompatible types Int64 and Int32) [TRT] [E] IBuilder::buildSerializedNetwork: Error Code 4: API Usage Error ((Unnamed Layer* 6) [Concatenation]: concat input tensors 0 and 2 have incompatible types Int64 and Int32) The error happened in "python/paddle/tensorrt/impls/common.py" because IConcatenationLayer requires all input to be same dtype. The (shape)tensor passed from paddle op will be int32, while the TRT IShapeLayer supports int64 shape; thereby, cannot be concatenated with each other. Here, we call `trt_shape` to get shape tensor whose dtype aligned with the dtype from paddle op.

test_converter_math can pass unittest with environment with TRT 10.6

- "pd_op.expand" - "pd_op.expand_as" - "pd_op.slice"

anderson101866 · 2024-12-05T11:35:35Z

OK重解conflict的時候沒有注意到，已上傳等待check重新驗證

anderson101866 · 2024-12-06T10:02:32Z

(format check已通過)
@YuanRisheng PR-CI-APPROVAL 這一項目前不明原因失敗，記得上次提到貌似因牽涉到paddle底層fluid，需要特別的人review approve，請在幫忙一下謝謝

vivienfanghuagood

LGTM

paddle-bot bot added the contributor External developers label Nov 19, 2024

anderson101866 marked this pull request as ready for review November 20, 2024 02:38

anderson101866 force-pushed the ameng/trt10_py_converter branch from e528354 to 252f83d Compare November 20, 2024 03:19

YuanRisheng previously approved these changes Nov 20, 2024

View reviewed changes

AdamzNV added the NVIDIA label Nov 21, 2024

onecatcn assigned vivienfanghuagood Nov 28, 2024

anderson101866 dismissed YuanRisheng’s stale review via a2ec045 December 5, 2024 06:24

anderson101866 force-pushed the ameng/trt10_py_converter branch 2 times, most recently from a2ec045 to 29ed847 Compare December 5, 2024 06:42

anderson101866 added 6 commits December 5, 2024 11:31

Add int64 in TRT->paddle dtype mapping function

4c5398b

Fix "test_converter_math" by enable op for TRT 10

09c8251

test_converter_math can pass unittest with environment with TRT 10.6

Adapt 3 manipulation converters to TRT 10

adb26c6

- "pd_op.expand" - "pd_op.expand_as" - "pd_op.slice"

anderson101866 force-pushed the ameng/trt10_py_converter branch from 29ed847 to adb26c6 Compare December 5, 2024 11:32

YuanRisheng approved these changes Dec 9, 2024

View reviewed changes

zhangbo9674 approved these changes Dec 9, 2024

View reviewed changes

vivienfanghuagood approved these changes Dec 9, 2024

View reviewed changes

YuanRisheng merged commit 25ed804 into PaddlePaddle:develop Dec 9, 2024
28 checks passed

Adapt python op converters covered in paddle-3-beta2 for TRT 10 #69510

Adapt python op converters covered in paddle-3-beta2 for TRT 10 #69510

Uh oh!

Conversation

anderson101866 commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

Uh oh!

paddle-bot bot commented Nov 19, 2024

Uh oh!

CLAassistant commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Nov 19, 2024

Uh oh!

YuanRisheng commented Nov 20, 2024

Uh oh!

anderson101866 commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YuanRisheng commented Nov 21, 2024

Uh oh!

anderson101866 commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YuanRisheng commented Nov 21, 2024

Uh oh!

anderson101866 commented Dec 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YuanRisheng commented Dec 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anderson101866 commented Dec 5, 2024

Uh oh!

YuanRisheng commented Dec 5, 2024

Uh oh!

anderson101866 commented Dec 5, 2024

Uh oh!

anderson101866 commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vivienfanghuagood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

anderson101866 commented Nov 19, 2024 •

edited

Loading

CLAassistant commented Nov 19, 2024 •

edited

Loading

anderson101866 commented Nov 21, 2024 •

edited

Loading

anderson101866 commented Nov 21, 2024 •

edited

Loading

anderson101866 commented Dec 3, 2024 •

edited

Loading

YuanRisheng commented Dec 3, 2024 •

edited

Loading

anderson101866 commented Dec 6, 2024 •

edited

Loading