Creating a recipe for CUDA-enabled pytorch

[odgaard]

Hi! Thank you for your incredible work with this layer.
After successfully building the python3-pytorch recipe today I discovered that it seems like a CPU-only install.
Have you made any attempts at getting a GPU-accelerated pytorch recipe working? Do you think it would be feasible to implement this based on your existing recipe?

[zboszor]

I assume you are building Yocto for x86-64 as I do.

Intel's MKL and oneDNN (MKLDNN) is already enabled in pytorch if meta-intel is added to the set of layers. This may make the GPU used implicitly, which is not the same as a fully accelerated build.

I haven't found a complete Yocto layer which provides CUDA with selectable version to provide support for obsolete hardware and is up to date with styhead and walnascar support.

Eventually I would want to go with chipStar (see #8 ) to fake CUDA which can work with any sufficiently mature OpenCL implementation. At least all its samples work with the Intel Compute Runtime's OpenCL driver. But chipStar is not there yet to build pytorch completely and you need Intel's icpx SYCL compiler from meta-intel to build chipStar's optional HIP libraries that may be needed to compile pytorch with chipStar's nvcc replacement. Unfortunately, icpx is not exposed as a native compiler executable by the Intel packages.

The XPU method of GPU acceleration in pytorch also needs icpx, so it's not enabled either.

So we're stuck with CPU-only at the moment. But at least the AVX2 and AVX512 stuff works.

FWIW, OpenVINO can be built using PACKAGECONFIG += "opencl" and it does provide GPU acceleration with a proper OpenCL driver.

[zboszor]

I have just seen the new AdaptiveCpp 25.02.0 release with "portable CUDA" which may be what we need for a "CUDA enabled" pytorch build using SYCL/OpenCL behind the scenes.

[cmcquinn]

I am interested in using this layer with meta-tegra. I might fork this layer, create some way to enable CUDA via a bbappend and then open a PR with my changes. Has anyone ever tried using this layer with meta-tegra before?

[zboszor]

I am interested in using this layer with meta-tegra. I might fork this layer, create some way to enable CUDA via a bbappend and then open a PR with my changes. Has anyone ever tried using this layer with meta-tegra before?

I have not used meta-tegra before but please, go ahead and try it.

See the python3-pytorch_%.bbappend for how it was done to support MKL when building for Intel.

You can add USE_CUDA=1 etc. in a meta-tegra-specific python3-pytorch_%.bbappend here under dynamic-layers/tegra and don't forget to change BBFILES_DYNAMIC in layer.conf.

[asterix]

@cmcquinn any luck with using this layer with meta-tegra and building pytorch with CUDA enabled? I'm trying to do the same. Currently, I can't even build pytorch due to broken python3-onnx dependencies in this layer (for arm64).

[cmcquinn]

@cmcquinn any luck with using this layer with meta-tegra and building pytorch with CUDA enabled? I'm trying to do the same. Currently, I can't even build pytorch due to broken python3-onnx dependencies in this layer (for arm64).

It turns out there the is already a recipe for pytorch in the meta-tegra-community repo which I ended up using.

[zboszor]

@cmcquinn any luck with using this layer with meta-tegra and building pytorch with CUDA enabled? I'm trying to do the same. Currently, I can't even build pytorch due to broken python3-onnx dependencies in this layer (for arm64).

FYI, the python3-onnx 1.20.0rc1 recipe builds just fine now and a lot of recipes were also fixed, including pytorch.

There are only two recipes that don't build:

• mediamtx, because it builds a rpicamdownloader binary and tries to run it on the host (doesn't work when the CPU arch is different)
• python3-kornia-rs, due to a rust 1.90 bug (but it builds fine for x86-64)

[asterix]

@zboszor Thanks! I should've mentioned more details in my comment. I'll do so now. My issues are all on scarthgap because that's the stable version I need to stick with for various reasons. If the forward fixes are portable to scarthgap, that'd be amazing because scarthgap is the latest LTS along with kernel 6.6, also being LTS.

A lot of these python packages' recipes are useful as ML tools, which are quite often hosted on aarch64 + Nvidia GPU platforms. Training and edge inferencing both.

Here's the protobuf linking error when building 1.16.2 python3-onnx.

Built target gen_onnx_data_proto
| [ 22%] Built target onnx_proto
| [ 97%] Built target onnx
| [ 98%] Linking CXX shared module onnx_cpp2py_export.cpython-312-aarch64-linux-gnu.so
| /yocto/nvidia/orin-nano/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/python3-onnx/1.16.2/recipe-sysroot-native/usr/bin/aarch64-oe4t-linux/../../libexec/aarch64-oe4t-linux/gcc/aarch64-oe4t-linux/13.4.0/ld: /yocto/nvidia/orin-nano/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/python3-onnx/1.16.2/recipe-sysroot-native/usr/lib/libprotobuf.so: error adding symbols: file in wrong format
| collect2: error: ld returned 1 exit status
| make[2]: *** [CMakeFiles/onnx_cpp2py_export.dir/build.make:169: onnx_cpp2py_export.cpython-312-aarch64-linux-gnu.so] Error 1
| make[1]: *** [CMakeFiles/Makefile2:223: CMakeFiles/onnx_cpp2py_export.dir/all] Error 2
| make: *** [Makefile:136: all] Error 2

@cmcquinn Yep, meta-tegra-community recipe is what I'm looking at too. I'm not quite on whinlatter, but on scarthgap, so I have to copy out from that layer although I use that layer for other recipes. pytorch + CUDA is the "desired" combination for any Nvidia platform at it seems insane dependency management hell even with Nvidia's own Jetpack SDK 6.2.1 + Pytorch installation instructions. That's why I turned here, but meta-python-ai didn't look optioned for Nvidia runtime OOB.

[zboszor]

@zboszor Thanks! I should've mentioned more details in my comment. I'll do so now. My issues are all on scarthgap because that's the stable version I need to stick with for various reasons. If the forward fixes are portable to scarthgap, that'd be amazing because scarthgap is the latest LTS along with kernel 6.6, also being LTS.

A lot of these python packages' recipes are useful as ML tools, which are quite often hosted on aarch64 + Nvidia GPU platforms. Training and edge inferencing both.

Here's the protobuf linking error when building 1.16.2 python3-onnx.

Built target gen_onnx_data_proto
| [ 22%] Built target onnx_proto
| [ 97%] Built target onnx
| [ 98%] Linking CXX shared module onnx_cpp2py_export.cpython-312-aarch64-linux-gnu.so
| /yocto/nvidia/orin-nano/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/python3-onnx/1.16.2/recipe-sysroot-native/usr/bin/aarch64-oe4t-linux/../../libexec/aarch64-oe4t-linux/gcc/aarch64-oe4t-linux/13.4.0/ld: /yocto/nvidia/orin-nano/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/python3-onnx/1.16.2/recipe-sysroot-native/usr/lib/libprotobuf.so: error adding symbols: file in wrong format
| collect2: error: ld returned 1 exit status
| make[2]: *** [CMakeFiles/onnx_cpp2py_export.dir/build.make:169: onnx_cpp2py_export.cpython-312-aarch64-linux-gnu.so] Error 1
| make[1]: *** [CMakeFiles/Makefile2:223: CMakeFiles/onnx_cpp2py_export.dir/all] Error 2
| make: *** [Makefile:136: all] Error 2

I'll try to fix the scarthgap build, just as the master/whinlatter branches are fixed for aarch64 now.