Dependency package metadata fetched excessively despite valid build cache

[engnatha]

🐞 bug report

Affected Rule

The issue is caused when dependencies are used with `pip.parse` (or `pip_parse` in WORKSPACE).

Is this a regression?

Yes, this previously worked well up to version `1.3.0`. It first broke with `1.4.0-rc0` and is still broken on the latest release of `1.5.1`.

Description

A clear and concise description of the problem...

pip.parse appears to invalidate the cache of third party pypi packages. Specifically I see a frequent rebuild of Fetching module extension pip in @@rules_python~//python/extensions:pip.bzl; starting reported in the beginning of builds despite nothing changing followed by fetches of all the packages again. Without an internet connection, this step fails and successive runs with internet restored keep failing until bazel shutdown is called. Note that this cache invalidation is not happening due to a terminated or failed build nor is it because any digest inputs have changed. It appears related to the lifetime of the bazel server.

🔬 Minimal Reproduction

See https://github.com/engnatha/rules_python_package_fetch_repro

🌍 Your Environment

Operating System:

  
Ubuntu 22.04 and 24.04
  

Output of bazel version:

  
Bazelisk version: v1.17.0
Build label: 7.5.0
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Thu Jan 30 18:33:17 2025 (1738261997)
Build timestamp: 1738261997
Build timestamp as int: 1738261997
  

Rules_python version:

  
1.4.0-rc0 and forward
  

[aignas]

This is expected side effect of solving #2434. Since we are not writing data to the lock file anymore, bzlmod relies on storing the extension metadata and it will have to refetch everything if you stop bazel build too early whilst it has not finished evaluating the extension fully.

Hope the explanation makes sense.

[engnatha]

Thanks for the quick reply. I'm definitely not caught up on the intricacies of bazelmod. However, this is a rather unsatisfying answer from a user perspective. Even when I change nothing about the source code or dependencies, this means periodically paying a high network overhead cost to rerun programs that should be reliably cached.

I also don't understand why this problem doesn't occur when the index url is omitted. If I embed the index url in the requirements lock file, then rules_python still fetches dependencies from my index as expected. The only difference is that things stay reliably cached even after shutting down the bazel server.

[engnatha]

And to clarify another thing, this isn't after prematurely stopping the build. The reproduction steps involve shutting down the bazel server after the build has completed and the binary has been executed successfully.

[aignas]

I also don't understand why this problem doesn't occur when the index url is omitted. If I embed the index url in the requirements lock file, then rules_python still fetches dependencies from my index as expected. The only difference is that things stay reliably cached even after shutting down the bazel server

rules_python is only fetching the file lists from the indexes. This should be relatively cheap. The actual wheels are only fetched if one of the following changes:

• rules_python code
• the requirements.txt file
• the env vars that affect the evaluation change

You can enable RULES_PYTHON_REPO_DEBUG to better understand what is going on. See docs.

And to clarify another thing, this isn't after prematurely stopping the build. The reproduction steps involve shutting down the bazel server after the build has completed and the binary has been executed successfully.

I think then I need to have more context into what is happening. Are environment variables leaking into the repository phase? I have not observed this on any other setups, so I do think that there is something unusual about your particular setup.

Since this is an experimental feature, as much diagnostic information as possible would be good to have here.

[engnatha]

Sure, I can try to include some better reproduction details to show our setup.

[engnatha]

Okay, I did some more investigation on this today and discovered a few interesting things.

• This is not specific to experimental_index_url. The same behavior is observed without that setting applied.
• The packages that complain are not related to what is used in the repository's dependencies. They are some constant list that must be used by internal tooling.
• Running the py_binary does not succeed after this error occurs even if internet access is restored. Another bazel shutdown is needed to correct things.

A reproduction for this is available in https://github.com/engnatha/rules_python_package_fetch_repro with steps I followed in the README.

Since this isn't specific to experimental_index_url, would it be appropriate to update the issue details with my findings or should we spin off a separate one?

[aignas]

Yes please, could you please update the original issue description?

[engnatha]

All updated! I took some time to walk back through previous releases and I'm seeing this work well in 1.3.0, so that should help narrow the search. I'll poke through some commits as well to see if anything jumps out.

[engnatha]

#2695 looks relevant

[engnatha]

Would it be appropriate to remove the need:repro label on this?