Skip to content

Conversation

konstin
Copy link
Member

@konstin konstin commented Aug 10, 2025

Add an incompatibility that lets pubgrub skip of marker packages when the base package already has an incompatible version to improve the error messages (#15199).

The change is also a small perf improvement. Overall this should be able to improve performance in slow cases by avoiding trying proxy package versions that are impossible anyway, for a (ideally very small cost) for tracking the additional incompatibility and tracking the base package for each proxy package.

$ hhyperfine --warmup 2 "uv pip compile --universal scripts/requirements/airflow.in" "target/release/uv pip compile --universal scripts/requirements/airflow.in"
Benchmark 1: uv pip compile --universal scripts/requirements/airflow.in
  Time (mean ± σ):     145.5 ms ±   3.9 ms    [User: 154.7 ms, System: 140.7 ms]
  Range (min … max):   139.2 ms … 153.4 ms    20 runs
 
Benchmark 2: target/release/uv pip compile --universal scripts/requirements/airflow.in
  Time (mean ± σ):     128.7 ms ±   5.5 ms    [User: 141.9 ms, System: 137.3 ms]
  Range (min … max):   121.8 ms … 142.0 ms    23 runs
 
Summary
  target/release/uv pip compile --universal scripts/requirements/airflow.in ran
    1.13 ± 0.06 times faster than uv pip compile --universal scripts/requirements/airflow.in

This implementation is the basic version: When we see a proxy foo{...}>=x,<y we add a dependency edge foo{...}>=x,<y -> foo>=x,<y. There are several way to extend this, which likely help more with performance than with error messages.

One idea is that if we see foo{...}>=x,<y but we already made a selection for foo==z outside that range, we can insert a dependency foo{...}!=z -> foo!=z. This avoids trying any version of the proxy package except the version that matches our previous selection.

Another is that if we see a dependency foo>=x,<y, we also add foo{...}>=x,y -> foo>=x,<y. This allows backtracking beyond foo immediately if all version of foo{...}>=x,<y are incompatible, since foo{...}>=x,<y incompatible -> foo>=x,<y incompatible -> the package that depended of foo>=x,<y is incompatible.

The cost for each of these operations is tracking an additional incompatibility per virtual package. An alternative approach is to only add the incompatibility lazily, only when we've tried several version of the virtual package already. This needs to be weighed of with the better error messages that the incompatibility gives, we unfortunately have only few large reference examples.

Requires astral-sh/pubgrub#45

Closes #15199

@konstin konstin added the error messages Messaging when something goes wrong label Aug 10, 2025
@konstin konstin temporarily deployed to uv-test-registries August 10, 2025 11:28 — with GitHub Actions Inactive
@konstin konstin temporarily deployed to uv-test-registries August 10, 2025 11:54 — with GitHub Actions Inactive
@konstin konstin force-pushed the konsti/add-incompat-for-proxy-packages branch from a289813 to 1774762 Compare August 27, 2025 16:58
@konstin konstin temporarily deployed to uv-test-registries August 27, 2025 17:00 — with GitHub Actions Inactive
@konstin konstin force-pushed the konsti/add-incompat-for-proxy-packages branch from 1774762 to 18e913d Compare August 27, 2025 17:04
@konstin konstin temporarily deployed to uv-test-registries August 27, 2025 17:07 — with GitHub Actions Inactive
@konstin konstin force-pushed the konsti/add-incompat-for-proxy-packages branch from 18e913d to 62a47ce Compare August 27, 2025 18:33
@konstin konstin temporarily deployed to uv-test-registries August 27, 2025 18:35 — with GitHub Actions Inactive
@konstin konstin requested review from BurntSushi and zanieb and removed request for BurntSushi August 27, 2025 19:00
@konstin konstin added the performance Potential performance improvement label Aug 27, 2025
@konstin konstin requested a review from BurntSushi August 27, 2025 19:00
@konstin
Copy link
Member Author

konstin commented Aug 27, 2025

Tagging both zanie and andrew because in an older version I broke a conflict test by accidentally including dependency groups, we should ensure that there are no other gotchas with dependency groups and/or conflicts.

@konstin konstin marked this pull request as ready for review August 27, 2025 19:01
@konstin konstin force-pushed the konsti/add-incompat-for-proxy-packages branch from 5b498fb to e434f41 Compare August 27, 2025 20:24
@konstin konstin temporarily deployed to uv-test-registries August 27, 2025 20:27 — with GitHub Actions Inactive
@konstin konstin force-pushed the konsti/add-incompat-for-proxy-packages branch from e434f41 to dd2fb4a Compare August 28, 2025 10:09
@konstin konstin temporarily deployed to uv-test-registries August 28, 2025 10:12 — with GitHub Actions Inactive
Copy link
Member

@zanieb zanieb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. I do not have a deep understanding of the implications here — it makes sense to have another reviewer.

Copy link
Member

@BurntSushi BurntSushi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes sense to me! This is a nice improvement. :-)

----- stderr -----
× No solution found when resolving dependencies:
╰─▶ Because only datasets<=2.18.0 is available and your project depends on datasets>=2.19, we can conclude that your project's requirements are unsatisfiable.
╰─▶ Because your project depends on datasets<2.19 and datasets>=2.19, we can conclude that your project's requirements are unsatisfiable.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice!

and pandas==1.5.3 depends on numpy{python_full_version >= '3.10'}>=1.21.0, we can conclude that pandas==1.5.3 depends on numpy>=1.21.0.
And because your project depends on numpy==1.20.3 and pandas==1.5.3, we can conclude that your project's requirements are unsatisfiable.
╰─▶ Because pandas==1.5.3 depends on numpy{python_full_version >= '3.10'}>=1.21.0 and your project depends on numpy==1.20.3, we can conclude that your project and pandas==1.5.3 are incompatible.
And because your project depends on pandas==1.5.3, we can conclude that your project's requirements are unsatisfiable.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much better!

Attempt at #15199 by adding an incompatibility that lets pubgrub skip of marker packages when the base package already has an incompatible version.

There are some false positives we still need to figure out, specifically around conflicts.

The change looks perf neutral for the one case I tried:

```
$ hyperfine --warmup 2 "target/release/uv pip compile --universal scripts/requirements/airflow.in" "target/release/uv pip compile --universal scripts/requirements/airflow.in"
  Benchmark 1: target/release/uv pip compile --universal scripts/requirements/airflow.in
    Time (mean ± σ):     482.5 ms ±  12.8 ms    [User: 602.7 ms, System: 205.2 ms]
    Range (min … max):   468.7 ms … 503.2 ms    10 runs

  Benchmark 2: target/release/uv pip compile --universal scripts/requirements/airflow.in
    Time (mean ± σ):     479.3 ms ±  21.0 ms    [User: 604.0 ms, System: 200.5 ms]
    Range (min … max):   464.1 ms … 537.5 ms    10 runs

    Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

  Summary
    target/release/uv pip compile --universal scripts/requirements/airflow.in ran
      1.01 ± 0.05 times faster than target/release/uv pip compile --universal scripts/requirements/airflow.in
```
@konstin konstin force-pushed the konsti/add-incompat-for-proxy-packages branch from dd2fb4a to a894509 Compare September 21, 2025 20:23
@konstin konstin temporarily deployed to uv-test-registries September 21, 2025 20:26 — with GitHub Actions Inactive
@konstin
Copy link
Member Author

konstin commented Sep 22, 2025

No changes in any tested package.

@konstin konstin merged commit a6daab4 into main Sep 22, 2025
159 checks passed
@konstin konstin deleted the konsti/add-incompat-for-proxy-packages branch September 22, 2025 11:26
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Sep 24, 2025
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.8.17` -> `0.8.22` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.8.22`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0822)

[Compare Source](astral-sh/uv@0.8.21...0.8.22)

Released on 2025-09-23.

##### Python

- Upgrade Pyodide to 0.28.3 ([#&#8203;15999](astral-sh/uv#15999))

##### Security

- Upgrade `astral-tokio-tar` to 0.5.5 which [hardens tar archive extraction](GHSA-3wgq-wrwc-vqmv) ([#&#8203;16004](astral-sh/uv#16004))

### [`v0.8.21`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0821)

[Compare Source](astral-sh/uv@0.8.20...0.8.21)

Released on 2025-09-23.

##### Enhancements

- Refresh lockfile when `--refresh` is provided ([#&#8203;15994](astral-sh/uv#15994))

##### Preview features

Add support for S3 request signing ([#&#8203;15925](astral-sh/uv#15925))

### [`v0.8.20`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0820)

[Compare Source](astral-sh/uv@0.8.19...0.8.20)

Released on 2025-09-22.

##### Enhancements

- Add `--force` flag for `uv cache clean` ([#&#8203;15992](astral-sh/uv#15992))
- Improve resolution errors with proxied packages ([#&#8203;15200](astral-sh/uv#15200))

##### Preview features

- Allow upgrading pre-release versions of the same minor Python version ([#&#8203;15959](astral-sh/uv#15959))

##### Bug fixes

- Hide `freethreaded+debug` Python downloads in `uv python list` ([#&#8203;15985](astral-sh/uv#15985))
- Retain the cache lock and temporary caches during `uv run` and `uvx` ([#&#8203;15990](astral-sh/uv#15990))

##### Documentation

- Add `package` level conflicts to the conflicting dependencies docs ([#&#8203;15963](astral-sh/uv#15963))
- Document pyodide support ([#&#8203;15962](astral-sh/uv#15962))
- Document support for free-threaded and debug Python versions ([#&#8203;15961](astral-sh/uv#15961))
- Expand the contribution docs on issue selection ([#&#8203;15966](astral-sh/uv#15966))
- Tweak title for viewing version in project guide ([#&#8203;15964](astral-sh/uv#15964))

### [`v0.8.19`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0819)

[Compare Source](astral-sh/uv@0.8.18...0.8.19)

Released on 2025-09-19.

##### Python

- Add CPython 3.14.0rc3
- Upgrade OpenSSL to 3.5.3

See the [python-build-standalone release notes](https://github.com/astral-sh/python-build-standalone/releases/tag/20250918) for more details.

##### Bug fixes

- Make `uv cache clean` parallel process safe ([#&#8203;15888](astral-sh/uv#15888))
- Fix implied `platform_machine` marker for `win_arm64` platform tag ([#&#8203;15921](astral-sh/uv#15921))

### [`v0.8.18`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0818)

[Compare Source](astral-sh/uv@0.8.17...0.8.18)

Released on 2025-09-17.

##### Enhancements

- Add PyG packages to torch backend ([#&#8203;15911](astral-sh/uv#15911))
- Add handling for unnamed conda environments in base environment detection ([#&#8203;15681](astral-sh/uv#15681))
- Allow selection of debug build interpreters ([#&#8203;11520](astral-sh/uv#11520))
- Improve `uv init` defaults for native build backend cache keys ([#&#8203;15705](astral-sh/uv#15705))
- Error when `pyproject.toml` target does not exist for dependency groups ([#&#8203;15831](astral-sh/uv#15831))
- Infer check URL from publish URL when known ([#&#8203;15886](astral-sh/uv#15886))
- Support Gitlab CI/CD as a trusted publisher ([#&#8203;15583](astral-sh/uv#15583))
- Add GraalPy 25.0.0 with support for Python 3.12 ([#&#8203;15900](astral-sh/uv#15900))
- Add `--no-clear` to `uv venv` to disable removal prompts ([#&#8203;15795](astral-sh/uv#15795))
- Add conflict detection between `--only-group` and `--extra` flags ([#&#8203;15788](astral-sh/uv#15788))
- Allow `[project]` to be missing from a `pyproject.toml` ([#&#8203;14113](astral-sh/uv#14113))
- Always treat conda environments named `base` and `root` as base environments ([#&#8203;15682](astral-sh/uv#15682))
- Improve log message when direct build for `uv_build` is skipped ([#&#8203;15898](astral-sh/uv#15898))
- Log when the cache is disabled ([#&#8203;15828](astral-sh/uv#15828))
- Show pyx organization name after authenticating ([#&#8203;15823](astral-sh/uv#15823))
- Use `_CONDA_ROOT` to detect Conda base environments ([#&#8203;15680](astral-sh/uv#15680))
- Include blake2b hash in `uv publish` upload form ([#&#8203;15794](astral-sh/uv#15794))
- Fix misleading debug message when removing environments in `uv sync` ([#&#8203;15881](astral-sh/uv#15881))

##### Deprecations

- Deprecate `tool.uv.dev-dependencies` ([#&#8203;15469](astral-sh/uv#15469))
- Revert "feat(ci): build loongarch64 binaries in CI ([#&#8203;15387](astral-sh/uv#15387))" ([#&#8203;15820](astral-sh/uv#15820))

##### Preview features

- Propagate preview flag to client for `native-auth` feature ([#&#8203;15872](astral-sh/uv#15872))
- Store native credentials for realms with the https scheme stripped ([#&#8203;15879](astral-sh/uv#15879))
- Use the root index URL when retrieving credentials from the native store ([#&#8203;15873](astral-sh/uv#15873))

##### Bug fixes

- Fix `uv sync --no-sources` not switching from editable to registry installations ([#&#8203;15234](astral-sh/uv#15234))
- Avoid display of an empty string when a path is the working directory ([#&#8203;15897](astral-sh/uv#15897))
- Allow cached environment reuse with `@latest` ([#&#8203;15827](astral-sh/uv#15827))
- Allow escaping spaces in --env-file handling ([#&#8203;15815](astral-sh/uv#15815))
- Avoid ANSI codes in debug! messages ([#&#8203;15843](astral-sh/uv#15843))
- Improve BSD tag construction ([#&#8203;15829](astral-sh/uv#15829))
- Include SHA when listing lockfile changes ([#&#8203;15817](astral-sh/uv#15817))
- Invert the logic for determining if a path is a base conda environment ([#&#8203;15679](astral-sh/uv#15679))
- Load credentials for explicit members when lowering ([#&#8203;15844](astral-sh/uv#15844))
- Re-add `triton` as a torch backend package ([#&#8203;15910](astral-sh/uv#15910))
- Respect `UV_INSECURE_NO_ZIP_VALIDATION=1` in duplicate header errors ([#&#8203;15912](astral-sh/uv#15912))

##### Documentation

- Add GitHub Actions to PyPI trusted publishing example ([#&#8203;15753](astral-sh/uv#15753))
- Add Coiled integration documentation ([#&#8203;14430](astral-sh/uv#14430))
- Add verbose output to the getting help section ([#&#8203;15915](astral-sh/uv#15915))
- Document `NO_PROXY` support ([#&#8203;15816](astral-sh/uv#15816))
- Document cache-keys for native build backends ([#&#8203;15811](astral-sh/uv#15811))
- Add documentation for dependency group `requires-python` ([#&#8203;14282](astral-sh/uv#14282))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS4xMTUuNiIsInVwZGF0ZWRJblZlciI6IjQxLjEyNS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
error messages Messaging when something goes wrong performance Potential performance improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Non-base packages don't collapse versions in resolution conflict traces.
3 participants