Skip to content

Conversation

ibraheemdev
Copy link
Member

Summary

Adds support for the functional TypedDict syntax, e.g.

Person = TypedDict("Person", { "name": str })

person: Person = { "name": "..." }

Part of astral-sh/ty#154.

@ibraheemdev ibraheemdev added the ty Multi-file analysis & type inference label Oct 7, 2025
@ibraheemdev ibraheemdev requested a review from dcreager as a code owner October 7, 2025 03:57
Copy link
Contributor

github-actions bot commented Oct 7, 2025

Diagnostic diff on typing conformance tests

Changes were detected when running ty on typing conformance tests
--- old-output.txt	2025-10-08 00:11:18.065496583 +0000
+++ new-output.txt	2025-10-08 00:11:21.376504168 +0000
@@ -1,6 +1,7 @@
 WARN ty is pre-release software and not ready for production use. Expect to encounter bugs, missing features, and fatal errors.
 fatal[panic] Panicked at /home/runner/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/29ab321/src/function/execute.rs:217:25 when checking `/home/runner/work/ruff/ruff/typing/conformance/tests/aliases_type_statement.py`: `PEP695TypeAliasType < 'db >::value_type_(Id(cc17)): execute: too many cycle iterations`
-fatal[panic] Panicked at /home/runner/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/29ab321/src/function/execute.rs:217:25 when checking `/home/runner/work/ruff/ruff/typing/conformance/tests/aliases_typealiastype.py`: `infer_definition_types(Id(16432)): execute: too many cycle iterations`
+fatal[panic] Panicked at /home/runner/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/29ab321/src/function/execute.rs:217:25 when checking `/home/runner/work/ruff/ruff/typing/conformance/tests/typeddicts_required.py`: `infer_definition_types(Id(13458)): execute: too many cycle iterations`
+fatal[panic] Panicked at /home/runner/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/29ab321/src/function/execute.rs:217:25 when checking `/home/runner/work/ruff/ruff/typing/conformance/tests/aliases_typealiastype.py`: `infer_definition_types(Id(16c32)): execute: too many cycle iterations`
 _directives_deprecated_library.py:15:31: error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `int`
 _directives_deprecated_library.py:30:26: error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `str`
 _directives_deprecated_library.py:36:41: error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `Self@__add__`
@@ -839,9 +840,11 @@
 tuples_type_form.py:15:1: error[invalid-assignment] Object of type `tuple[Literal[1], Literal[""]]` is not assignable to `tuple[int, int]`
 tuples_type_form.py:25:1: error[invalid-assignment] Object of type `tuple[Literal[1]]` is not assignable to `tuple[()]`
 tuples_type_form.py:36:1: error[invalid-assignment] Object of type `tuple[Literal[1], Literal[2], Literal[3], Literal[""]]` is not assignable to `tuple[int, ...]`
+typeddicts_alt_syntax.py:23:44: error[invalid-argument-type] Argument is incorrect: Expected `_TypedDictSchema`, found `dict[Unknown | str, Unknown | <class 'str'>]`
 typeddicts_operations.py:37:20: error[missing-typed-dict-key] Missing required key 'name' in TypedDict `Movie` constructor
 typeddicts_operations.py:62:1: error[unresolved-attribute] Type `MovieOptional` has no attribute `clear`
 typeddicts_readonly.py:24:4: error[invalid-assignment] Cannot assign to key "members" on TypedDict `Band`: key is marked read-only
+typeddicts_readonly.py:36:4: error[invalid-assignment] Cannot assign to key "members" on TypedDict `Band2`: key is marked read-only
 typeddicts_readonly.py:50:4: error[invalid-assignment] Cannot assign to key "title" on TypedDict `Movie1`: key is marked read-only
 typeddicts_readonly.py:51:4: error[invalid-assignment] Cannot assign to key "year" on TypedDict `Movie1`: key is marked read-only
 typeddicts_readonly.py:60:4: error[invalid-assignment] Cannot assign to key "title" on TypedDict `Movie2`: key is marked read-only
@@ -849,12 +852,11 @@
 typeddicts_readonly_inheritance.py:36:4: error[invalid-assignment] Cannot assign to key "name" on TypedDict `Album2`: key is marked read-only
 typeddicts_readonly_inheritance.py:65:19: error[missing-typed-dict-key] Missing required key 'name' in TypedDict `RequiredName` constructor
 typeddicts_type_consistency.py:69:21: error[invalid-key] Invalid key access on TypedDict `A3`: Unknown key "y"
-typeddicts_type_consistency.py:101:1: error[invalid-assignment] Object of type `Unknown | None` is not assignable to `str`
 typeddicts_type_consistency.py:126:56: error[invalid-argument-type] Invalid argument to key "inner_key" with declared type `str` on TypedDict `Inner1`: value of type `Literal[1]`
 typeddicts_usage.py:23:7: error[invalid-key] Invalid key access on TypedDict `Movie`: Unknown key "director"
 typeddicts_usage.py:24:17: error[invalid-assignment] Invalid assignment to key "year" with declared type `int` on TypedDict `Movie`: value of type `Literal["1982"]`
 typeddicts_usage.py:28:17: error[missing-typed-dict-key] Missing required key 'name' in TypedDict `Movie` constructor
 typeddicts_usage.py:28:18: error[invalid-key] Invalid key access on TypedDict `Movie`: Unknown key "title"
 typeddicts_usage.py:40:24: error[invalid-type-form] The special form `typing.TypedDict` is not allowed in type expressions. Did you mean to use a concrete TypedDict or `collections.abc.Mapping[str, object]` instead?
-Found 857 diagnostics
+Found 859 diagnostics
 WARN A fatal error occurred while checking some files. Not all project files were analyzed. See the diagnostics list above for details.

Comment on lines 5613 to 5616
else {
// Emit a diagnostic here? We seem to support non-string literals.
unimplemented!()
};
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be erroring here? pyright doesn't seem to support string constants as TypedDict keys, but we currently ignore non-literal keys when type-checking.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From https://typing.python.org/en/latest/spec/typeddict.html#use-of-final-values-and-literal-types:

Type checkers are only expected to support actual string literals, not final names or literal types, for specifying keys in a TypedDict type definition

Copy link
Member

@AlexWaygood AlexWaygood Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Only expected to support" is not the same as "must emit an error for anything else", though...

In general, we defer a lot of operations to type-check time that other type checkers attempt to do during (their equivalent of) semantic indexing. That puts limitations on other type checkers that we don't necessarily have for things like this, which is why language like this appears in the spec about what type checkers are "expected" to support

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the code to ignore these for now. I think eventually it makes sense to at least warn if we don't support them.

Copy link
Contributor

@sharkdp sharkdp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fantastic — thank you very much.

// understand a more specific meta type in order to correctly handle `__getitem__`.
match self {
TypedDictType::FromClass(class) => SubclassOfType::from(db, class),
TypedDictType::Synthesized(_) => KnownClass::TypedDictFallback.to_class_literal(db),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I'm wondering why this doesn't cause problems for dunder-calls on synthesized TypedDicts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KnownInstanceType::TypedDictType is special-cased in Type::bindings, but I think it would otherwise go through instance_fallback to an instance of KnownClass::TypedDictFallback, not meta_type to its class literal.

Comment on lines 5613 to 5616
else {
// Emit a diagnostic here? We seem to support non-string literals.
unimplemented!()
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From https://typing.python.org/en/latest/spec/typeddict.html#use-of-final-values-and-literal-types:

Type checkers are only expected to support actual string literals, not final names or literal types, for specifying keys in a TypedDict type definition

@ibraheemdev ibraheemdev force-pushed the ibraheem/typed-dict-constructor branch from 646c631 to d2cb5b8 Compare October 7, 2025 17:29
@ibraheemdev ibraheemdev force-pushed the ibraheem/typed-dict-constructor branch from d2cb5b8 to 2959ff1 Compare October 7, 2025 17:37
Copy link
Contributor

github-actions bot commented Oct 7, 2025

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Comment on lines 6870 to 6871
/// A single instance of `typing.TypedDict`.
TypedDictType(TypedDictType<'db>),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I haven't read through this whole PR yet so I'm not sure what the correct doc-comment here should be, but I don't think this can be an accurate description of what TypedDictType represents... TypedDict is a function at runtime, so it is impossible to create an "instance of TypedDict". Calling TypedDict "as a function" at runtime creates a new class in exactly the same way as inheriting from TypedDict at runtime:

>>> import typing
>>> type(typing.TypedDict)
<class 'function'>
>>> X = typing.TypedDict("X", {})
>>> X
<class '__main__.X'>
>>> type(X)
<class 'typing._TypedDictMeta'>
>>> class Y(typing.TypedDict): ...
... 
>>> Y
<class '__main__.Y'>
>>> type(Y)
<class 'typing._TypedDictMeta'>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems incorrect to me that we infer different types for X and Y (and different types for X.__class__ and Y.__class__) here on your PR branch. They're both class objects in exactly the same way at runtime:

from typing import TypedDict, reveal_type

X = TypedDict("X", {})

reveal_type(X)  # revealed: typing.TypedDict
reveal_type(X.__class__)  # revealed: TypedDictFallback

class Y(TypedDict): ...

reveal_type(Y)  # revealed: <class 'Y'>
reveal_type(Y.__class__)  # revealed: type

We previously experimented with making the Type::ClassLiteral() variant internally wrap an enum, so that we could express the fact that not all class-literal objects at runtime are created via class statements. #19998 turned out not to be the correct approach for NewTypes (calls to NewType don't actually create classes!), but I think it could be the right approach here, and for the functional enum.Enum syntax, and for collections.namedtuple() calls, and for the functional typing.NamedTuple syntax, and for three-argument calls to type(), since those all do actually create classes at runtime.

Having said all that, this would all complicate your approach (possibly by quite a lot), so I'm okay if you don't want to try making Type::ClassLiteral an enum. I only mention it because I think we'll have the same issue of "class objects created via function calls" for all these other things we'll need to support in the long term, too. So it may be worth digging in and trying to pull off that refactor at some point.

Copy link
Member Author

@ibraheemdev ibraheemdev Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I see. Maybe I misread this comment, which I thought meant that we don't need to support definitionless classes for functional TypedDicts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I think that was just my oversight in that comment, not thinking about the fact that functional TypedDict does actually create a class at runtime.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't speak for Carl, but I think possibly what he was saying there was that it turned out that NewType calls don't actually create new class objects afterall, so actually NewType is different from functional NamedTuple/enum/TypedDict/classes created using three-argument type(). Meaning that in fact, #20126 doesn't help us here, because unlike #19998, it doesn't propose turning ClassLiteral into an enum

Copy link
Member

@AlexWaygood AlexWaygood Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, I posted my reply before I saw Carl's!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, that makes a lot of sense. I would prefer to get this PR merged even if it is slightly incorrect (assuming it doesn't generate a lot of ecosystem false positives), but I'm happy to continue this work to support the ClassLiteral refactor. It probably makes sense to do that with functional TypedDicts first, rather than implement a new feature along with the refactor.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That SGTM, breaking it up into chunks will make it easier to review too!

Copy link
Contributor

github-actions bot commented Oct 7, 2025

mypy_primer results

Changes were detected when running on open source projects
isort (https://github.com/pycqa/isort)
- isort/output.py:535:25: error[invalid-argument-type] Argument to function `import_statement` is incorrect: Expected `Sequence[str]`, found `@Todo | None | list[Unknown]`
+ isort/output.py:535:25: error[invalid-argument-type] Argument to function `import_statement` is incorrect: Expected `Sequence[str]`, found `Any | None | list[Unknown]`
- isort/output.py:545:25: error[invalid-argument-type] Argument to function `import_statement` is incorrect: Expected `Sequence[str]`, found `@Todo | None | list[Unknown]`
+ isort/output.py:545:25: error[invalid-argument-type] Argument to function `import_statement` is incorrect: Expected `Sequence[str]`, found `Any | None | list[Unknown]`
- isort/output.py:553:29: error[invalid-argument-type] Argument to function `import_statement` is incorrect: Expected `Sequence[str]`, found `@Todo | None | list[Unknown]`
+ isort/output.py:553:29: error[invalid-argument-type] Argument to function `import_statement` is incorrect: Expected `Sequence[str]`, found `Any | None | list[Unknown]`

graphql-core (https://github.com/graphql-python/graphql-core)
- tests/utilities/test_build_client_schema.py:682:42: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- Found 424 diagnostics
+ Found 423 diagnostics

dragonchain (https://github.com/dragonchain/dragonchain)
+ dragonchain/transaction_processor/level_4_actions_utest.py:73:73: error[invalid-argument-type] Invalid argument to key "dc_id" with declared type `str` on TypedDict `L1Headers`: value of type `Literal[123]`
+ dragonchain/transaction_processor/level_4_actions_utest.py:73:90: error[invalid-argument-type] Invalid argument to key "block_id" with declared type `str` on TypedDict `L1Headers`: value of type `Literal[124]`
- Found 315 diagnostics
+ Found 317 diagnostics

mypy (https://github.com/python/mypy)
+ mypy/typeshed/stdlib/logging/config.pyi:45:38: error[unsupported-operator] Operator `|` is unsupported between objects of type `typing.TypedDict` and `<class 'dict[str, Any]'>`
- Found 1834 diagnostics
+ Found 1835 diagnostics

artigraph (https://github.com/artigraph/artigraph)
+ src/arti/types/python.py:258:13: error[invalid-argument-type] Argument is incorrect: Expected `_TypedDictSchema`, found `dict[@Todo, @Todo]`
- Found 146 diagnostics
+ Found 147 diagnostics

operator (https://github.com/canonical/operator)
+ ops/model.py:85:51: error[invalid-type-form] Variable of type `Literal["_SettableStatusName | _ReadOnlyStatusName"]` is not allowed in a type expression
- ops/pebble.py:637:58: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- ops/pebble.py:727:58: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- ops/pebble.py:796:58: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
+ ops/pebble.py:2110:23: error[invalid-argument-type] Invalid argument to key "access" with declared type `Literal["untrusted", "metrics", "read", "admin"]` on TypedDict `IdentityDict`: value of type `str`
- Found 100 diagnostics
+ Found 99 diagnostics

meson (https://github.com/mesonbuild/meson)
+ mesonbuild/cargo/manifest.py:268:28: error[invalid-key] Invalid key access on TypedDict `Package`: Unknown key "package"
+ mesonbuild/cargo/manifest.py:347:17: error[invalid-key] Invalid key access on TypedDict `BuildTarget`: Unknown key "path" - did you mean "name"?
+ mesonbuild/cargo/manifest.py:362:17: error[invalid-key] Invalid key access on TypedDict `BuildTarget`: Unknown key "path" - did you mean "name"?
+ mesonbuild/cargo/manifest.py:376:17: error[invalid-key] Invalid key access on TypedDict `BuildTarget`: Unknown key "path" - did you mean "name"?
+ mesonbuild/cargo/manifest.py:390:17: error[invalid-key] Invalid key access on TypedDict `BuildTarget`: Unknown key "path" - did you mean "name"?
- Found 889 diagnostics
+ Found 894 diagnostics

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/cli/deployment.py:392:48: error[invalid-assignment] Invalid assignment to key "anchor_date" with declared type `str` on TypedDict `IntervalScheduleOptions`: value of type `datetime`
+ src/prefect/cli/root.py:121:48: error[invalid-key] Invalid key access on TypedDict `VersionInfo`: Unknown key "full-revisionid"
+ src/prefect/utilities/dockerutils.py:67:46: error[invalid-key] Invalid key access on TypedDict `VersionInfo`: Unknown key "full-revisionid"
- Found 3198 diagnostics
+ Found 3201 diagnostics

scikit-build-core (https://github.com/scikit-build/scikit-build-core)
+ src/scikit_build_core/build/_wheelfile.py:51:22: error[no-matching-overload] No overload of function `field` matches arguments
- Found 52 diagnostics
+ Found 53 diagnostics

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/test/unit/test_type_clinic.py:197:39: error[invalid-argument-type] Argument is incorrect: Expected `_TypedDictSchema`, found `dict[str, <class 'int'> | <class 'float'> | <class 'str'>]`
+ static_frame/test/unit/test_type_clinic.py:198:39: error[invalid-argument-type] Argument is incorrect: Expected `_TypedDictSchema`, found `dict[str, <class 'int'> | <class 'float'> | <class 'bool'>]`
- Found 1888 diagnostics
+ Found 1890 diagnostics
No memory usage changes detected ✅

```py
fields = {"name": str}

# error: [invalid-argument-type] "Argument is incorrect: Expected `_TypedDictSchema`, found `dict[Unknown | str, Unknown | <class 'str'>]`"
Copy link
Member Author

@ibraheemdev ibraheemdev Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how we should display TypedDictSchema in diagnostics, given that it is an internal type. Ideally we would have a special diagnostic for this specific case, but I believe the TypedDictSchema type might still show up in the IDE in other cases.

Copy link
Contributor

github-actions bot commented Oct 7, 2025

ecosystem-analyzer results

Lint rule Added Removed Changed
invalid-argument-type 6 0 3
invalid-key 7 0 0
unused-ignore-comment 0 4 0
invalid-assignment 1 0 0
invalid-type-form 1 0 0
unsupported-operator 1 0 0
Total 16 4 3

Full report with detailed diff (timing results)

@ibraheemdev
Copy link
Member Author

ibraheemdev commented Oct 7, 2025

The main limitation of this PR based on the ecosystem report looks to be inheriting from a TypedDict created with the functional syntax:

error: [invalid-base] Invalid class base with type `typing.TypedDict` 

The errors are a bit unfortunate, but I'm not sure there's an easy way to avoid these false positives before the ClassLiteral refactor.

@AlexWaygood
Copy link
Member

The main limitation of this PR based on the ecosystem report looks to be inheriting from a TypedDict created with the functional syntax:

error: [invalid-base] Invalid class base with type `typing.TypedDict` 

The errors are a bit unfortunate, but I'm not sure there's an easy way to avoid these false positives before the ClassLiteral refactor.

Could you add a branch to ClassBase::try_from_type similar to this one, to silence the false positives?

SpecialFormType::Callable => Self::try_from_type(
db,
todo_type!("Support for Callable as a base class"),
subclass,
),

@ibraheemdev ibraheemdev force-pushed the ibraheem/typed-dict-constructor branch from 36411dd to ac2ed3e Compare October 7, 2025 21:19
@ibraheemdev ibraheemdev force-pushed the ibraheem/typed-dict-constructor branch from ac2ed3e to 26c25be Compare October 7, 2025 21:21
Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking on board the feedback -- this looks great!

@AlexWaygood
Copy link
Member

+ static_frame/test/unit/test_type_clinic.py:197:39: error[invalid-argument-type] Argument is incorrect: Expected `_TypedDictSchema`, found `dict[str, <class 'int'> | <class 'float'> | <class 'str'>]`

It would be nice if we could special-case this kind of diagnostic to say something like "Expected a dictionary literal with string-literal keys and types as values", rather than exposing the internal _TypedDictSchema type here. But we can defer that; this seems fine for an initial implementation

@ibraheemdev
Copy link
Member Author

It looks like there's a panic with recursive TypeDict definitions in the typing conformance suite:

RecursiveMovie = TypedDict(
    "RecursiveMovie", {"title": Required[str], "predecessor": NotRequired["RecursiveMovie"]}
)

@carljm
Copy link
Contributor

carljm commented Oct 8, 2025

It looks like there's a panic with recursive TypeDict definitions in the typing conformance suite

Shoot, I should have thought of this in advance. I think it means that we will have to make inference of the actual dict spec lazy, similar to how it is for class typed-dicts by default (since all our understanding of class bodies is lazy).

It may be that the best way to do this is similar to what I've done for TypeVar definitions in #20598, where we more fully special-case the entire assignment statement. This would probably eliminate the need for some of the machinery added in this PR around integrating with the full call-binding machinery.

@ibraheemdev ibraheemdev marked this pull request as draft October 8, 2025 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ecosystem-analyzer ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants