Document improvements to build-isolation setups

[charliermarsh]

Summary

There's a lot here (maybe it should go somewhere else?), but this is a complex topic and I wanted to include a lot of copy-pasteable examples for common cases.

Closes #10694.
Closes #13959.
Closes #15248.
Closes #15316.

[vwxyzjn]

Love the examples! The flash attention one doesn't work, but I tested deep_ep and deep_gemm which all works!

[vwxyzjn]

This doesn't work:

ubuntu@costa-dev-worker-0:~/code/thirdparty/test_uv$ cat pyproject.toml
[project]
name = "project"
version = "0.1.0"
description = "..."
readme = "README.md"
requires-python = ">=3.12"
dependencies = ["flash-attn", "torch"]

[tool.uv.extra-build-dependencies]
flash-attn = [{ dependencies = ["torch"], match-runtime = true }]

[tool.uv.extra-build-variables]
flash-attn = { FLASH_ATTENTION_SKIP_CUDA_BUILD = "TRUE" }
ubuntu@costa-dev-worker-0:~/code/thirdparty/test_uv$ uv sync
warning: Failed to parse `pyproject.toml` during settings discovery:
  TOML parse error at line 10, column 14
     |
  10 | flash-attn = [{ dependencies = ["torch"], match-runtime = true }]
     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  data did not match any variant of untagged enum ExtraBuildDependencyWire

error: Failed to parse: `pyproject.toml`
  Caused by: TOML parse error at line 10, column 14
   |
10 | flash-attn = [{ dependencies = ["torch"], match-runtime = true }]
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
data did not match any variant of untagged enum ExtraBuildDependencyWire

[charliermarsh]

Oh thank you for trying this, there’s a typo 😩

[vwxyzjn]

I have put a minimal repro here: https://github.com/vwxyzjn/minimal-uv-deep-ep-gemm-installation/tree/main

[zanieb]

I have some minor edits and, as you said, we probably want to find a better home for all this content, but let's just get this in and iterate from there since it's so important.

[tobiasdiez]

These are very nice improvements. Thanks a lot for everyone that worked on this!

Another use case of "no-build-isolation" of the main project is that some build backends (like python-meson) allow to recompile files on the fly in editable installs. For this to work, naturally the build dependencies of the main project need to be available also at runtime. (More details at #10694 (comment))
It's not clear to me if those recent changes cover this use case as well. One still needs to create an "extra" for the build dependencies, duplicating what is already in build-system.requires, and then use uv sync --extra build, right? Or is there a way to sync the build requirements of the main project without duplicating them to an extra?

[charliermarsh]

Thanks @tobiasdiez! I can take a look and add some documentation for that. Can you give me a minimal pyproject.toml + a set of commands to reproduce, for testing?

[zsol]

This is a typo

Suggested change

	that changes to these settings will not trigger a reinstall and rebuild of the affected packages.
	that changes to these settings will trigger a reinstall and rebuild of the affected packages.

[charliermarsh]

PR!!!

[zsol]

#15390

[tobiasdiez]

Does this also ensure that the package annotated with match-runtime = true will only be built once? So in the example below (assuming that torch wouldn't ship wheels), would torch be built once as the build dependency of deepspeed and then another time to be installed in the main env? Or does uv cache the built of torch and reuses it?

I hope it is cached - in this case, match-runtime might be also a very useful option to set for packages that take a long time to build. I can open a PR to add this if wished.

[tobiasdiez]

Thanks @tobiasdiez! I can take a look and add some documentation for that. Can you give me a minimal pyproject.toml + a set of commands to reproduce, for testing?

Sorry for taking so long to come back to you. Here is an example pyproject.toml that shows the problem:

[project]
name = "test"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []
[build-system]
build-backend = 'mesonpy'
requires = ["cython>=3.1.3", "meson-python>=0.18.0"]

This is just what you get with uv init plus adding the meson-python as a build backend. To run uv sync you also need a simple meson.build, say

project(
  'test',
  ['c', 'cpp', 'cython'],
  meson_version: '>=1.2',
)

py_module = import('python')
py = py_module.find_installation(pure: false)

py.install_sources(
  'main.py',
)

In a real application you would now actually use cython to compile an extension etc, but for the sake of having a simple example let's ignore that.

After uv sync the package is installed in editable mode, and in particular you have a file .venv\Lib\site-packages\_test_editable_loader.py which intercepts all imports and rebuilds the project if necessary. This means you need to have the build dependencies also installed at runtime. This is however not the case currently if you run uv sync.

Current workaround is:

uv pip install meson-python cython
uv sync --no-build-isolation

So it's similar to the problems discussed in this PR, but the difference is that the no-build-isolation concerns the main project and not one of its dependencies.

Let me know if this is sufficient to understand the problem, or if you would prefer to have a full example that actually compiles something etc.