Skip to content

Conversation

AlexWaygood
Copy link
Member

Summary

Further work towards #14169.

This adjusts our handling of * imports so that we pay attention to visibility constraints in the external module (the module symbols are being imported from) when we infer the boundness of symbols defined via a * import. I.e., for something like this:

```toml
python-version = "3.9"
```

`a.py`:

```py
import sys

if sys.version_info >= (3, 10):
    X = True
```

`b.py`:

```py
from a import *

print(X)
```

We will now correctly emit an [unresolved-reference] diagnostic on the use of X in b.py: the visibility constraints in a.py resolve to Truthiness::AlwaysFalse, so no symbols are actually bound in b.py as a result of the * import.

Implementation

When adding each *-import definition to the UseDefMap, we snapshot the SymbolState as it was prior to the * import definition (possibly) overriding any (possible) previous definitions. Then, in symbol::symbol_from_bindings_impl, we evaluate * import definitions to determine the boundness of the symbol in the external module. If the symbol is in fact unbound in the external module (due to visibility constraints resolving to Truthiness::AlwaysFalse), we determine that the * import did not in fact provide an additional binding for this symbol, and in the importing module we fall back to the SymbolState for that symbol prior to the *-import definition.

In order to be able to resolve * imports from symbol.rs as well as from types/infer.rs, several functions are pulled out of infer.rs into a new import_resolution module where they can be accessed from both types/infer.rs and symbol.rs.

Problems with this approach

This approach works well for symbols in the external module that are statically known to either be definitely bound or definitely unbound. But the logic in symbol.rs is still somewhat naive, and doesn't work for possibly-unbound definitions in the external module. I've added a failing test in this PR to illustrate the issue: I don't know how to fix it properly, and would welcome ideas. In addition to the new failing test, the PR does not address any of these TODOs, and ideally it would:

M, # TODO: could emit diagnostic about being possibly unbound
N, # TODO: could emit diagnostic about being possibly unbound
O, # TODO: could emit diagnostic about being possibly unbound
P, # TODO: could emit diagnostic about being possibly unbound
Q, # TODO: could emit diagnostic about being possibly unbound
R, # TODO: could emit diagnostic about being possibly unbound
S, # TODO: could emit diagnostic about being possibly unbound
T, # TODO: could emit diagnostic about being possibly unbound
U, # TODO: could emit diagnostic about being possibly unbound
V, # TODO: could emit diagnostic about being possibly unbound
W, # TODO: could emit diagnostic about being possibly unbound

Test Plan

Existing assertions in mdtests updated; new ones added.

@AlexWaygood AlexWaygood added the ty Multi-file analysis & type inference label Apr 7, 2025
Copy link
Contributor

github-actions bot commented Apr 7, 2025

mypy_primer results

No ecosystem changes detected ✅

@AlexWaygood AlexWaygood force-pushed the alex/star-statically-known-2 branch 2 times, most recently from 7b02daf to 06e8dc2 Compare April 7, 2025 20:25
Copy link
Contributor

@carljm carljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty good! Didn't do a full review, just commented on the main open issues I saw.

@AlexWaygood AlexWaygood force-pushed the alex/star-statically-known-2 branch from 06e8dc2 to 4c6b6d1 Compare April 8, 2025 09:31
@AlexWaygood
Copy link
Member Author

Quite apart from any of the big things I was stuck on (thank you all for your review comments!), I realised after sleeping on it that I could significantly simplify things by reviving the idea I had previously in #16923 (comment). There's now no need for moving any logic from infer.rs into a new import_resolution submodule, and all the questions about "Where do we do tracing?" are now redundant 🥳

@AlexWaygood AlexWaygood force-pushed the alex/star-statically-known-2 branch from eeb09bf to 3706dca Compare April 8, 2025 14:50
Copy link

codspeed-hq bot commented Apr 8, 2025

CodSpeed Performance Report

Merging #17286 will degrade performances by 12.49%

Comparing alex/star-statically-known-2 (9415521) with main (f1ba596)

Summary

❌ 1 regressions
✅ 31 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
red_knot_check_file[cold] 96.7 ms 110.5 ms -12.49%

@AlexWaygood
Copy link
Member Author

I tried out @sharkdp's suggested alternative implementation, and it appears to work beautifully... except that this PR is now triggering(?) a Salsa panic in some unrelated tests in a file I didn't touch:

  crates/red_knot_python_semantic/resources/mdtest/properties.md:203 panicked at /home/runner/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/296a8c7/src/tracked_struct.rs:832:21
  crates/red_knot_python_semantic/resources/mdtest/properties.md:203 access to field whilst the value is being initialized

which looks like the same Salsa panic that @dcreager was running into in #17023 before he applied ea12548 to that PR...

@AlexWaygood AlexWaygood force-pushed the alex/star-statically-known-2 branch 2 times, most recently from 534964f to 93bc320 Compare April 8, 2025 15:38
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All changes to this file are just commenting out the test that causes the Salsa panic (I just applied @dcreager's commit ea12548 to this branch 😆)

Copy link
Contributor

@carljm carljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so nice and clean now! Love it.

I think given our upcoming deadlines I would be inclined to merge this even with a significant regression (especially if that regression gets smaller percentage-wise when we check a larger codebase), and prefer to have someone spend dedicated time later looking for performance wins. But it wouldn't hurt to at least look at some profiles (the CodSpeed ones are sadly useless, so they have to be generated locally) to understand which queries we spend more time in now.

Copy link
Contributor

@sharkdp sharkdp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice — Thank you!

@AlexWaygood AlexWaygood force-pushed the alex/star-statically-known-2 branch from 93bc320 to 6ec2191 Compare April 9, 2025 14:15
@AlexWaygood
Copy link
Member Author

Merging #17286 will degrade performances by 12.49%

We discussed this internally. It's obviously far from ideal, but for now we're going to eat the performance regression. We have some ideas for how to address it, but they may or may not work out, and we want to move onto other tasks for now. I'll writeup a separate issue with our findings from looking at the tracing logs and various profiles.

@AlexWaygood AlexWaygood enabled auto-merge (squash) April 9, 2025 14:34
@AlexWaygood AlexWaygood merged commit 6ec4c6a into main Apr 9, 2025
20 checks passed
@AlexWaygood AlexWaygood deleted the alex/star-statically-known-2 branch April 9, 2025 14:36
dcreager added a commit that referenced this pull request Apr 9, 2025
* main:
  [red-knot] Allow explicit specialization of generic classes (#17023)
  [`airflow`] Refactor `AIR301` logic and fix typos (`AIR301`) (#17293)
  [`airflow`] Extract `AIR312` from `AIR302` rules (`AIR302`, `AIR312`) (#17152)
  [red-knot] Improve handling of visibility constraints in external modules when resolving `*` imports (#17286)
  [red-knot] Add more tests for `*` imports (#17315)
Glyphack pushed a commit to Glyphack/ruff that referenced this pull request Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ty Multi-file analysis & type inference
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants