The issue of Linux wheels slowing down

[ouco1986]

My graphics card is 9070XT, and I noticed that the Linux wheels seem to be slowing down on Comyfui? Especially with the addition of the -- use pytorch cross attention parameter, the speed will increase from 20it/s to 16it/s under sd1.5. The nightly version of PyTorch currently has a speed of 27it/s, which is currently the fastest in my testing.

[ouco1986]

Also, after adding the -- use cell cross cross attention parameter under Windows, the speed of sd1.5 is 21it/s

[zichguan-amd]

Hi @ouco1986, can you provide an example workload and more details about your ROCm and PyTorch versions?

[zichguan-amd]

It's likely due to #1408, currently the Linux wheels have aotriton disabled, see https://github.com/ROCm/TheRock/blob/main/external-builds/pytorch/README.md#project-and-feature-support-status

[ouco1986]

@zichguan-amd ，Thank you for your answer. After I installed version 2.71 of troch, the speed improved. After adding the export TORCHLOCM-AORITON-INABLEEXEPERIMENTAL=1 and the -- use-pytorch-cross-attention parameter to start. Speed restored to 25it/s

[ouco1986]

@zichguan-amd Additionally, I would like to share my experience on Windows and Linux. When using Comfyui Wan2.2 to generate videos, using block nodes to reduce video memory usage during inference can run smoothly. However, in the VAE stage, a large amount of video memory is required, resulting in OOM errors. Under Windows, there is a shared existing mechanism that does not terminate operations. There should also be a mechanism similar to shared video memory under Linux, but in reality, it does not work at all. I don't know what the reason is. Is it a problem with the Linux kernel or the Linux graphics card driver? What about issues such as Rocm, Troch, Comfyui? For this reason, the user experience of running Comfyui on Linux is completely inferior to that on Windows.

[waltercool]

Aotriton for FA is not supported on Linux, yet.

That causes some operations to take huge amount of memory, processing photos and videos with Pytorch may cause lots of OOM, depending of the model used.

[ouco1986]

Aotriton for FA is not supported on Linux, yet.

That causes some operations to take huge amount of memory, processing photos and videos with Pytorch may cause lots of OOM, depending of the model used.

The model itself is a problem, but the main issue now is that under Linux, when there is only a little bit of VRAM to run, it will not switch to shared VRAM, while Windows can, especially for some nodes that do not significantly affect speed. For example, even if VAE has a little bit in memory, it can ultimately complete the task. However, Linux makes room or hip errors just for a small amount of video memory. The previous time was completely wasted. In other words, workflows that can run under Windows are not compatible with Linux, and the difference is significant.

[zichguan-amd]

Currently on Linux HIP cannot allocate beyond the VRAM size, you can try using tiled VAE or use CPU VAE to reduce VRAM usage.

[ouco1986]

Currently on Linux HIP cannot allocate beyond the VRAM size, you can try using tiled VAE or use CPU VAE to reduce VRAM usage.

Thanks for the explanation. It looks like we'll just have to wait. Because VAE nodes can be processed in chunks, and I've already done that. But some nodes also reference VAE operations or other operations that cause OOM, and these cannot be processed in chunks.