[MIGraphX EP] [ROCm EP] Update CI to use ROCm 6.4

[TedThemistokleous]

Description

Update Onnxruntime CI to use latest ROCm release ROCm 6.3.2

Motivation and Context

Ensure CI is testing up to date with recent Onnxruntime. AMD validates changes based off the latest ROCm release and adds additional features on top to validate changes prior to being pushed to our internal testing branches and then up streamed to Microsoft/Onnxruntime:main . Right now Onnxruntime is testing things a few releases back for their CI

[snnn]

/azp run Big Models, Linux Android Emulator QNN CI Pipeline, Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline

[snnn]

/azp run Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline,

[snnn]

/azp run Windows GPU CUDA CI Pipeline, Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 7 pipeline(s).

[snnn]

/azp run Linux MIGraphX CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tianleiwu]

Need change the pipeline configuration:

onnxruntime/tools/ci_build/github/azure-pipelines/linux-migraphx-ci-pipeline.yml

Lines 39 to 42 in 28ee049

	- name: RocmVersion
	value: 6.2
	- name: RocmVersionPatchSuffix
	value: ".3"

Note that ROCm EP uses another docker file and yml file. It is fine to only upgrade MigraphX CI in this PR.

[TedThemistokleous]

Need change the pipeline configuration:

onnxruntime/tools/ci_build/github/azure-pipelines/linux-migraphx-ci-pipeline.yml

Lines 39 to 42 in 28ee049

	- name: RocmVersion
	value: 6.2
	- name: RocmVersionPatchSuffix
	value: ".3"

Note that ROCm EP uses another docker file and yml file. It is fine to only upgrade MigraphX CI in this PR.

Updated

[snnn]

/azp run Big Models, Linux Android Emulator QNN CI Pipeline, Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline

[snnn]

/azp run Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline,

[snnn]

/azp run Windows GPU CUDA CI Pipeline, Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

[snnn]

/azp run Linux MIGraphX CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 7 pipeline(s).

[tianleiwu]

/azp run Linux ROCm CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tianleiwu]

@TedThemistokleous, The ROCm CI Pipeline failed. It is because cupy does not support ROCm 6.3 yet. I think the solution is to remove cupy from the docker file, then set an environment variable KERNEL_EXPLORER_TEST_USE_CUPY=0. See the usage of cupy in test:

onnxruntime/onnxruntime/python/tools/kernel_explorer/kernels/utils.py

Lines 114 to 115 in d5338da

	if os.getenv("KERNEL_EXPLORER_TEST_USE_CUPY", "0") == "1":
	import cupy as cp

Updated: I sent a PR to update ROCm CI: #23577.

[tianleiwu]

@TedThemistokleous, #23577 was merged to main. Could you merge the main branch to this PR?

[TedThemistokleous]

@TedThemistokleous, #23577 was merged to main. Could you merge the main branch to this PR?

Sure let me gat that in. I've updated the CI to use the latest ROCm 6.4 as well

[TedThemistokleous]

@tianleiwu merge from main has been completed.

[tianleiwu]

/azp run Linux MIGraphX CI Pipeline, Linux ROCm CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[tianleiwu]

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).