configs: torch-2.9 xpu configs

[dvrogozh]

This PR starts cooking Pytorch 2.9 variant configs. At the moment:

1. Added XPU dependencies respective to 2.9 RC (https://download.pytorch.org/whl/test/xpu)
2. Added Nvidia configuration (thank you, @DEKHTIARJonathan)

This PR requires these versions of plugins:

• https://github.com/wheelnext/intel-variant-provider/releases/tag/v0.0.2-rc1 (or later of v0.0.2 series)
• https://github.com/wheelnext/nvidia-variant-provider/releases/tag/v0.0.2

CC: @atalman @DEKHTIARJonathan

[dvrogozh]

These are steps I used to verify pytorch xpu variant build (these are steps, not a working shell script as there are few missing parts which are assumed such as installed or built variantlib, variant-repack and intel-variant-provider):

LOCAL_INDEX=~/tmp/repack/xpu/
CONFIG=~/git/variant-repack/configs/torch-2.9/

mkdir $LOCAL_INDEX

# Download XPU wheels with dependencies:
# torchvision will download torch as a dependency, we need both
python3 -m pip download torchvision --index-url https://download.pytorch.org/whl/test/xpu --dest $LOCAL_INDEX

# Create directory structure to serve as --index-url
cd $LOCAL_INDEX
for pkg in `ls *.whl`; do name=${pkg%%-*}; name=${name//_/-}; mkdir $name && mv $pkg $name/; done
mv MarkupSafe/ markupsafe

# Move original torch away and repack with variants:
mv torch torch-no-variant && mkdir torch
mv torchvision torchvision-no-variant && mkdir torchvision

variant_repack build -i $(ls torch-no-variant/*xpu*.whl) -o ./torch/ --pyproject-toml $CONFIG/torch_pyproject.toml --variant-config-toml $CONFIG/torch_variant_config.toml --variant-config-name intel --metadata-config-name torch

variant_repack build -i $(ls torchvision-no-variant/*xpu*.whl) -o ./torchvision/ --pyproject-toml $CONFIG/torch_pyproject.toml --variant-config-toml $CONFIG/torch_variant_config.toml --variant-config-name intel --metadata-config-name torchvision

# That’s a better variant as it checks some errors in package configurations:
variantlib generate-index-json -d torch/
variantlib generate-index-json -d torchvision/
# or
cp $CONFIG/torch-2.9.0-variants.json torch/
cp $CONFIG/torchvision-2.9.0-variants.json torch/

# these are previously built with python3 -m build -w .
mkdir nvidia-variant-provider/ && cp ~/git/nvidia-variant-provider/dist/*.whl nvidia-variant-provider/
mkdir intel-variant-provider/ && cp ~/git/intel-variant-provider/dist/*.whl intel-variant-provider/

# Now we can serve the index:
python -m http.server 8000

And in other shell we can install (on Intel GPU compatible system this should install xpu pytorch variant):

1. With patched pip from https://github.com/wheelnext/pep_xxx_wheel_variants/tree/e44bf06c26872be158949f93fecb0c8ee762653e/pep_xxx_wheel_variants:

git clone https://github.com/wheelnext/pep_xxx_wheel_variants.git
cd pep_xxx_wheel_variants/pep_xxx_wheel_variants # note subfolder
python3 -m pip install .
python3 -m pip install --verbose --index-url http://localhost:8000/ torchvision

2. With patched uv from Add prototype implementation of wheel variant specification astral-sh/uv#12203:

git clone https://github.com/astral-sh/uv.git cd uv
git fetch origin pull/12203/head:pr_12203 && git checkout pr_12203
cargo run pip install torchvision --index http://localhost:8000/ --refresh --verbose

[DEKHTIARJonathan]

Alright I added NV config and rebased on master.

Let's wait a little bit - I'm confirming a few details

[dvrogozh]

@DEKHTIARJonathan, thank you for update.

[jithunnair-amd]

git clone https://github.com/wheelnext/pep_xxx_wheel_variants.git
cd pep_xxx_wheel_variants/pep_xxx_wheel_variants # note subfolder
python3 -m pip install .

@dvrogozh If I use the pip installation instructions from "1. With patched pip from", it installs from the pep-xxx-wheel-variants branch of https://github.com/wheelnext/pip.git:

Collecting pip@ git+https://github.com/wheelnext/pip.git@pep-xxx-wheel-variants (from pep-xxx-wheel-variants==1.0.0)
  Cloning https://github.com/wheelnext/pip.git (to revision pep-xxx-wheel-variants) to /tmp/pip-install-l6t93i3_/pip_d3807507f49b4d479cc044a1ee725eff
  Running command git clone --filter=blob:none --quiet https://github.com/wheelnext/pip.git /tmp/pip-install-l6t93i3_/pip_d3807507f49b4d479cc044a1ee725eff
  Running command git checkout -b pep-xxx-wheel-variants --track origin/pep-xxx-wheel-variants
  Switched to a new branch 'pep-xxx-wheel-variants'
  Branch 'pep-xxx-wheel-variants' set up to track remote branch 'pep-xxx-wheel-variants' from 'origin'.
  Resolved https://github.com/wheelnext/pip.git to commit 2616ad9bf1ee564af8ca1116956eec9438fb5821

Is that what we want though? It looks like the latest changes are being added to the pep-xxx-wheel-variants-dev branch.

Installing pip directly from that branch might be needed:

pip install --force-reinstall git+https://github.com/wheelnext/pip.git@pep-xxx-wheel-variants-dev
Collecting git+https://github.com/wheelnext/pip.git@pep-xxx-wheel-variants-dev
  Cloning https://github.com/wheelnext/pip.git (to revision pep-xxx-wheel-variants-dev) to /tmp/pip-req-build-h22vi44s
  Running command git clone --filter=blob:none --quiet https://github.com/wheelnext/pip.git /tmp/pip-req-build-h22vi44s
  Running command git checkout -b pep-xxx-wheel-variants-dev --track origin/pep-xxx-wheel-variants-dev
  Switched to a new branch 'pep-xxx-wheel-variants-dev'
  Branch 'pep-xxx-wheel-variants-dev' set up to track remote branch 'pep-xxx-wheel-variants-dev' from 'origin'.
  Resolved https://github.com/wheelnext/pip.git to commit 1a9ec589e5a1306f041e70d5be64c067b7dd4e77

That commit 1a9ec589e5a1306f041e70d5be64c067b7dd4e77 is the same commit for the pip submodule in https://github.com/wheelnext/pep_xxx_wheel_variants/

[dvrogozh]

@jithunnair-amd, that's what I have tested and what worked for me. As for whether that's what we want, I hope so since the decision was made to use the main branch as a driving point. I did not dive under the hood after that on what is being fetched from where. It might be that you have spotted a misconfiguration - submit and issue or ping folks in wheelnext on Slack. My personal opinion is that for the better consistency all branches except main and release branches should be completely pruned from wheelnext repos and folks should start create user forks to avoid a mess in the upstream (which is wheelnext).

[dvrogozh]

By the way, remember that pip is not a goal right now, uv is. On the moment I wrote a #4 (comment) there was only a branch with the changes we needed, but today a version of uv was released (I think that's RC for now). So you might wish to try with it:

curl -LsSf https://astral.sh/uv/install.sh | INSTALLER_DOWNLOAD_URL=https://wheelnext.astral.sh/v0.0.2 sh

[dvrogozh]

@DEKHTIARJonathan , why did not you list deps to remove here for Nvidia? did you miss them or there is a logic change anywhere for nvidia or in variant-repack?

[DEKHTIARJonathan]

Close - will re-open from a branch inside this repo