`--torch-backend=auto` ignores CUDA compute capability when installing `torch` for GTX 1080 Ti

[hamzaq2000]

Summary

Tested the following command on a system with a GTX 1080 Ti and CUDA 12.8 driver installed (what shows in nvidia-smi).

uv pip install torch --torch-backend=auto --preview

Then ran this test script:

import torch
tensor = torch.randn(3, 4, device='cuda')
print(tensor)

And got this:

  cpu = _conversion_method_template(device=torch.device("cpu"))
/root/env/lib/python3.11/site-packages/torch/cuda/__init__.py:262: UserWarning:
    Found GPU0 NVIDIA GeForce GTX 1080 Ti which is of cuda capability 6.1.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability supported by this library is 7.5.

  warnings.warn(
/root/env/lib/python3.11/site-packages/torch/cuda/__init__.py:287: UserWarning:
NVIDIA GeForce GTX 1080 Ti with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_75 sm_80 sm_86 sm_90 sm_100 sm_120 compute_120.
If you want to use the NVIDIA GeForce GTX 1080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
Traceback (most recent call last):
  File "/root/torch_test.py", line 4, in <module>
    tensor = torch.randn(3, 4, device='cuda')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I think just looking at the installed CUDA version/driver might not be enough, the CUDA compute capability supported by different torch versions is of relevance as well. In this case, we have a GTX 1080 Ti with CUDA 12.8 drivers installed, but we can't actually use the cu128 torch wheel, which --torch-backend=auto installs, as it doesn't support GPUs with CUDA compute capability < 7.5.

Some relevant discussion I found:
https://discuss.pytorch.org/t/gpu-compute-capability-support-for-each-pytorch-version/62434/4
https://github.com/moi90/pytorch_compute_capabilities

I then also tested without --torch-backend=auto, so simply:

uv pip install torch

This seems to install the cu126 wheel (going off the PyTorch website), and with this the test script worked just fine, meaning the cu126 wheel has not yet dropped support for CUDA compute capability 6.1 devices.

Platform

Linux 5.15.0-142-generic x86_64 GNU/Linux

Version

0.8.0

Python version

Python 3.11.13

[DEKHTIARJonathan]

Unfortunately I'm not knowledgeable enough to comment on the feasibility of this, so I won't. But I thought it was worth putting out there; hopefully it is fixable and --torch-backend=auto can cleanly install the the correct torch wheel from the correct index for all NVIDIA GPUs.

Downgrade your driver to CUDA 12.6 and it will fix the problem.

Charlie may address it - unfortunately it's like chasing a rabbit and a lot of if/else because it will be on a per-release basis.
Make your life easy and downgrade the driver (12.8 won't benefit you in this case given that you must use 12.6 artifacts)

[charliermarsh]

I suspect downgrading the user-mode driver would still result in the same behavior with --torch-backend=auto since they would still be above the minimum required kernel driver for the PyTorch 2.8 wheels, right? Like, they'd still be on a kernel driver of above 525.60.13, so we'd still assume CUDA 12.8 compatibility.

[hamzaq2000]

@DEKHTIARJonathan

Downgrade your driver to CUDA 12.6 and it will fix the problem

As I said, all works fine if I just do uv pip install torch instead of uv pip install torch --torch-backend=auto --preview. I.e. keep the driver the same at 12.8, but just install the cu126 wheel instead of cu128. So getting it running is not an issue.

The issue is that I'd like it if --torch-backend=auto could do this instead of manual intervention. I'm writing a library in which it'd be nice to have just a single uv pip install to install it instead of multiple cases where the user has to first check the compute capability of their GPU.

The workaround I've come up with for now is just using the cu128 wheel outright. I'll just tell the user to make sure to have a CUDA 12 driver and a GPU with compute capability >= 7.5. For users that have a CUDA 12 driver but a GPU with compute capability < 7.5, they can run:

uv pip install library[nvidia-old]

and this will install the cu126 wheel.

[charliermarsh]

Yeah it would be nice if auto handled this. I think it's possible (likely easier if we use nvml as Jonathan suggested elsewhere).

[hamzaq2000]

This is just me speculating, but would it also not be required to keep a table of each torch wheel and the CUDA compute capabilities it supports? For example, the error message I got said the following (cu128 wheel):

NVIDIA GeForce GTX 1080 Ti with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_75 sm_80 sm_86 sm_90 sm_100 sm_120 compute_120.

Or perhaps there is some link between driver version, CUDA version, and torch wheels that I am unaware of, sidestepping the need to check for compute capability altogether.

[DEKHTIARJonathan]

@charliermarsh you can completely disregard the KMD. It's extremely rare that it has any impact anywhere.

So rare we couldn't find one package anywhere in the entire Python ecosystem that actually has a dependency on the KMD. Consequently we removed this feature from the NV variant plugin to avoid confusing people.

Just focus on libcuda.so and you'll get the right answer you look for 🤘

[DEKHTIARJonathan]

Use this https://github.com/wheelnext/nvidia-variant-provider/blob/dev/nvidia_variant_provider/detect_cuda.py#L31

Simple and you can directly dlopen the library.

We use pynvml here for simplicity but you can access nvml directly from rust.

[DEKHTIARJonathan]

This is just me speculating, but would it also not be required to keep a table of each torch wheel and the CUDA compute capabilities it supports? For example, the error message I got said the following (cu128 wheel):

NVIDIA GeForce GTX 1080 Ti with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_75 sm_80 sm_86 sm_90 sm_100 sm_120 compute_120.

Or perhaps there is some link between driver version, CUDA version, and torch wheels that I am unaware of, sidestepping the need to check for compute capability altogether.

Yes in theory... But what about the usecases where people have different GPUs on the machine... And that would mean updating and maintaining that table for a very long time.

Also SM compatibility rules are very complicated. There's PTX/SASS compat, jit compilation, and a whole lot of family oriented compatibility. These rules are quite complicated to write if you don't want them hyper restrictive.

I'll give you an example. Torch has never been built AFAIK for SM 6.1 (GTX 1080) yet it works perfectly. So if there was a filtering on just "SM 60" like it's built... You would not get anything. Yet built with SM 6.0 doesn't mean compatible with the whole 6.x family... Complicated 😅

[hamzaq2000]

Wow, the rabbit hole goes deep. TIL 😅