[ty] typecheck dict methods for `TypedDict` #19874

PrettyWood · 2025-08-12T09:43:06Z

Summary

Typecheck get(), setdefault(), pop() for TypedDict

from typing import TypedDict
from typing_extensions import NotRequired

class Employee(TypedDict):
    name: str
    department: NotRequired[str]

emp = Employee(name="Alice", department="Engineering")

emp.get("name")
emp.get("departmen", "Unknown")
emp.pop("department")
emp.pop("name")

part of astral-sh/ty#154

Test Plan

Updated Markdown tests

github-actions · 2025-08-25T12:53:44Z

Diagnostic diff on typing conformance tests

Changes were detected when running ty on typing conformance tests

--- old-output.txt	2025-08-29 13:40:04.333301009 +0000
+++ new-output.txt	2025-08-29 13:40:06.963314808 +0000
@@ -849,7 +849,6 @@
 tuples_type_form.py:25:1: error[invalid-assignment] Object of type `tuple[Literal[1]]` is not assignable to `tuple[()]`
 tuples_type_form.py:36:1: error[invalid-assignment] Object of type `tuple[Literal[1], Literal[2], Literal[3], Literal[""]]` is not assignable to `tuple[int, ...]`
 typeddicts_operations.py:37:5: error[missing-typed-dict-key] Missing required key 'name' in TypedDict `Movie` constructor
-typeddicts_operations.py:60:1: error[type-assertion-failure] Argument does not have asserted type `str | None`
 typeddicts_operations.py:62:1: error[unresolved-attribute] Type `MovieOptional` has no attribute `clear`
 typeddicts_readonly_inheritance.py:65:1: error[missing-typed-dict-key] Missing required key 'name' in TypedDict `RequiredName` constructor
 typeddicts_type_consistency.py:69:21: error[invalid-key] Invalid key access on TypedDict `A3`: Unknown key "y"
@@ -859,5 +858,5 @@
 typeddicts_usage.py:28:1: error[missing-typed-dict-key] Missing required key 'name' in TypedDict `Movie` constructor
 typeddicts_usage.py:28:18: error[invalid-key] Invalid key access on TypedDict `Movie`: Unknown key "title"
 typeddicts_usage.py:40:24: error[invalid-type-form] The special form `typing.TypedDict` is not allowed in type expressions. Did you mean to use a concrete TypedDict or `collections.abc.Mapping[str, object]` instead?
-Found 860 diagnostics
+Found 859 diagnostics
 WARN A fatal error occurred while checking some files. Not all project files were analyzed. See the diagnostics list above for details.

github-actions · 2025-08-25T12:55:55Z

`mypy_primer` results

Changes were detected when running on open source projects

openlibrary (https://github.com/internetarchive/openlibrary)
+ openlibrary/plugins/worksearch/code.py:447:34: error[invalid-argument-type] Argument to function `__new__` is incorrect: Expected `Iterable[Unknown]`, found `list[str] | None`
+ openlibrary/plugins/worksearch/code.py:447:61: error[invalid-argument-type] Argument to function `__new__` is incorrect: Expected `Iterable[Unknown]`, found `list[str] | None`
- Found 709 diagnostics
+ Found 711 diagnostics

discord.py (https://github.com/Rapptz/discord.py)
- discord/activity.py:882:38: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- discord/member.py:390:41: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- Found 533 diagnostics
+ Found 531 diagnostics

meson (https://github.com/mesonbuild/meson)
+ mesonbuild/interpreter/interpreter.py:1804:69: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `dict[str, str] | None`, found `dict[OptionKey, str | int | list[str]]`
+ mesonbuild/modules/python.py:169:42: error[invalid-key] Invalid key access on TypedDict `ExtensionModuleKw`: Unknown key "limited_api"
- Found 808 diagnostics
+ Found 810 diagnostics

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/runner/storage.py:851:13: error[invalid-return-type] Return type does not match returned value: expected `str`, found `str | (Any & ~AlwaysFalsy & ~Secret[Unknown]) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
+ src/prefect/runner/storage.py:858:13: error[invalid-return-type] Return type does not match returned value: expected `str`, found `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | str | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
+ src/prefect/runner/storage.py:859:16: warning[possibly-unbound-attribute] Attribute `startswith` on type `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | (str & ~AlwaysFalsy) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)` is possibly unbound
+ src/prefect/runner/storage.py:866:13: error[invalid-return-type] Return type does not match returned value: expected `str`, found `str | (Any & ~AlwaysFalsy & ~Secret[Unknown]) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
+ src/prefect/runner/storage.py:867:20: warning[possibly-unbound-attribute] Attribute `startswith` on type `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | (str & ~AlwaysFalsy) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)` is possibly unbound
+ src/prefect/runner/storage.py:872:12: error[invalid-return-type] Return type does not match returned value: expected `str`, found `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | (str & ~AlwaysFalsy) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
- Found 3013 diagnostics
+ Found 3019 diagnostics

dd-trace-py (https://github.com/DataDog/dd-trace-py)
+ ddtrace/llmobs/_experiment.py:506:28: warning[redundant-cast] Value is already of type `int`
- Found 6602 diagnostics
+ Found 6603 diagnostics

No memory usage changes detected ✅

github-actions · 2025-08-25T13:01:08Z

`ecosystem-analyzer` results

Lint rule	Added	Removed
`invalid-return-type`	4	0
`invalid-argument-type`	3	0
`possibly-unbound-attribute`	2	0
`unused-ignore-comment`	0	2
`invalid-key`	1	0
`redundant-cast`	1	0
Total	11	2

sharkdp

Thank you very much. This looks great.

I looked through a few ecosystem changes here (and found a problem with .get calls with a default argument of a different type, which I commented on inline), but it might be worth going through a few more to see if we're missing anything in the implementation.

crates/ty_python_semantic/src/types/class.rs

crates/ty_python_semantic/src/types/infer.rs

sharkdp · 2025-08-29T11:06:21Z

I'm currently looking into making the required changes here.

crates/ty_python_semantic/src/types.rs

dcreager

Just a couple of comments re the generics/typevar changes.

dcreager · 2025-08-29T13:06:01Z

crates/ty_python_semantic/src/types/class.rs

+                            )),
+                        )
+                    }))
+                    .collect();


nit: CallableSignature::from_overloads takes in an iterator, so I think you can skip the collect here

Oops, yes. I missed this in my review. Thanks.

crates/ty_python_semantic/src/types/class.rs

crates/ty_python_semantic/src/types/generics.rs

sharkdp · 2025-08-29T13:19:47Z

Ok, after adding some fallback overloads for .get, we're now down to a reasonable number of ecosystem changes.

discord.py (https://github.com/Rapptz/discord.py)
- discord/activity.py:882:38: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- discord/member.py:390:41: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- Found 533 diagnostics
+ Found 531 diagnostics

Correct, they are .pop ing a required key, and silence that with # type: ignore.

openlibrary (https://github.com/internetarchive/openlibrary)
+ openlibrary/plugins/worksearch/code.py:447:34: error[invalid-argument-type] Argument to function `__new__` is incorrect: Expected `Iterable[Unknown]`, found `list[str] | None`
+ openlibrary/plugins/worksearch/code.py:447:61: error[invalid-argument-type] Argument to function `__new__` is incorrect: Expected `Iterable[Unknown]`, found `list[str] | None`
- Found 709 diagnostics
+ Found 711 diagnostics

True positives. They have a key with value type list[str] | None, and then use .get("that_key", []) in an attempt to replace None with []. But that doesn't work. get won't replace values of type None with the default. It only replaces missing keys with the default.

meson (https://github.com/mesonbuild/meson)
+ mesonbuild/interpreter/interpreter.py:1804:69: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `dict[str, str] | None`, found `dict[OptionKey, str | int | list[str]]`
+ mesonbuild/modules/python.py:169:42: error[invalid-key] Invalid key access on TypedDict `ExtensionModuleKw`: Unknown key "limited_api"
- Found 808 diagnostics
+ Found 810 diagnostics

Both look like true positives to me

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/runner/storage.py:851:13: error[invalid-return-type] Return type does not match returned value: expected `str`, found `str | (Any & ~AlwaysFalsy & ~Secret[Unknown]) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
+ src/prefect/runner/storage.py:858:13: error[invalid-return-type] Return type does not match returned value: expected `str`, found `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | str | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
+ src/prefect/runner/storage.py:859:16: warning[possibly-unbound-attribute] Attribute `startswith` on type `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | (str & ~AlwaysFalsy) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)` is possibly unbound
+ src/prefect/runner/storage.py:866:13: error[invalid-return-type] Return type does not match returned value: expected `str`, found `str | (Any & ~AlwaysFalsy & ~Secret[Unknown]) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
+ src/prefect/runner/storage.py:867:20: warning[possibly-unbound-attribute] Attribute `startswith` on type `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | (str & ~AlwaysFalsy) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)` is possibly unbound
+ src/prefect/runner/storage.py:872:12: error[invalid-return-type] Return type does not match returned value: expected `str`, found `(Any & ~AlwaysFalsy & ~Secret[Unknown]) | (str & ~AlwaysFalsy) | (Secret[str] & ~AlwaysFalsy & ~Secret[Unknown]) | (@Todo(Type::Intersection.call()) & ~AlwaysFalsy)`
- Found 3013 diagnostics
+ Found 3019 diagnostics

Seems unrelated to TypedDict, only surfaces because we don't infer Unknown anymore. Potentially related to astral-sh/ty#456.

dd-trace-py (https://github.com/DataDog/dd-trace-py)
+ ddtrace/llmobs/_experiment.py:506:28: warning[redundant-cast] Value is already of type `int`
- Found 6602 diagnostics
+ Found 6603 diagnostics

True positive

sharkdp

Thanks again!

crates/ty_python_semantic/src/types/class.rs

AlexWaygood · 2025-08-29T13:42:15Z

crates/ty_python_semantic/resources/mdtest/typed_dict.md

+    # error: [invalid-argument-type] "Cannot pop required field 'name' from TypedDict `Person`"
+    reveal_type(p.pop("name"))  # revealed: Unknown


I feel like ideally we'd still infer str here (despite the fact that the operation is illegal, it seems pretty clear that the result of this operation will always be a str), but I can see that that might be difficult to do consistently with the synthesized-overload approach. And consistently inferring Unknown here seems better than sometimes inferring Unknown, sometimes str (depending on exactly how the method is invoked on the typeddict object).

I feel like ideally we'd still infer str here (despite the fact that the operation is illegal, it seems pretty clear that the result of this operation will always be a str)

Yes, interesting point. I agree that this would probably be the better return type, but as you said, it's not possible to achieve that with synthesized overloads, unless we manage to synthesize overloads that trigger specific CallErrors when selected, or similar. Until then, Unknown is not ideal, but it's also not wrong. And if you're going to # type: ignore your illegal .pop operation, you might as well throw in a cast or a redeclaration :-)

* main: (28 commits) [ty] `__class_getitem__` is a classmethod (#20192) [ty] Support `__init_subclass__` (#20190) Update dependency ruff to v0.12.11 (#20184) Update rui314/setup-mold digest to 725a879 (#20181) Update CodSpeedHQ/action action to v3.8.1 (#20183) Update cargo-bins/cargo-binstall action to v1.15.3 (#20182) Update Rust crate camino to v1.1.12 (#20185) Update Rust crate clap to v4.5.46 (#20186) Update Rust crate mimalloc to v0.1.48 (#20187) [ty] Sync vendored typeshed stubs (#20188) [ty] improve cycle-detection coverage for apply_type_mapping (#20159) [ty] don't assume that deferred type inference means deferred name resolution (#20160) Less confidently mark f-strings as empty when inferring truthiness (#20152) [ty] skip a slow seed in fuzzer (#20161) Revert "[ty] Use `invalid-assignment` error code for invalid assignments to `ClassVar`s" (#20158) [ty] add six ecosystem projects to good.txt (#20157) [ty] Use `invalid-assignment` error code for invalid assignments to `ClassVar`s (#20156) [ty] minor TypedDict fixes (#20146) [ty] ensure union normalization really normalizes (#20147) [ty] typecheck dict methods for `TypedDict` (#19874) ...

## Summary Typecheck `get()`, `setdefault()`, `pop()` for `TypedDict` ```py from typing import TypedDict from typing_extensions import NotRequired class Employee(TypedDict): name: str department: NotRequired[str] emp = Employee(name="Alice", department="Engineering") emp.get("name") emp.get("departmen", "Unknown") emp.pop("department") emp.pop("name") ``` <img width="838" height="529" alt="Screenshot 2025-08-12 at 11 42 12" src="https://github.com/user-attachments/assets/77ce150a-223c-4931-b914-551095d8a3a6" /> part of astral-sh/ty#154 ## Test Plan Updated Markdown tests --------- Co-authored-by: David Peter <[email protected]>

PrettyWood requested review from AlexWaygood, MichaReiser, carljm, dcreager and sharkdp as code owners August 12, 2025 09:43

AlexWaygood added the ty Multi-file analysis & type inference label Aug 12, 2025

carljm removed their request for review August 19, 2025 00:23

PrettyWood added 3 commits August 25, 2025 14:48

feat: overloads for TypedDict.get()

463970d

feat: add pop() and setdefault()

cc4e3e0

feat: add clean error for .pop() on required field

3902667

sharkdp force-pushed the ty-typeddict-methods branch from 771f1d1 to 3902667 Compare August 25, 2025 12:51

sharkdp added the ecosystem-analyzer label Aug 25, 2025

sharkdp requested changes Aug 25, 2025

View reviewed changes

sharkdp added 2 commits August 29, 2025 11:49

Merge remote-tracking branch 'origin/main' into ty-typeddict-methods

2539329

Remove comment about rule

84d2a21

sharkdp added ecosystem-analyzer and removed ecosystem-analyzer labels Aug 29, 2025

sharkdp reviewed Aug 29, 2025

View reviewed changes

crates/ty_python_semantic/src/types.rs Show resolved Hide resolved

Synthesize generic .get and .pop methods

476cc3e

sharkdp force-pushed the ty-typeddict-methods branch from 4626add to 476cc3e Compare August 29, 2025 12:01

sharkdp added ecosystem-analyzer and removed ecosystem-analyzer labels Aug 29, 2025

Fallback overloads for .get

f3fc071

sharkdp added ecosystem-analyzer and removed ecosystem-analyzer labels Aug 29, 2025

dcreager reviewed Aug 29, 2025

View reviewed changes

sharkdp added 2 commits August 29, 2025 15:28

feedback

59dbf5b

Merge remote-tracking branch 'origin/main' into ty-typeddict-methods

0e493b9

sharkdp approved these changes Aug 29, 2025

View reviewed changes

AlexWaygood reviewed Aug 29, 2025

View reviewed changes

crates/ty_python_semantic/src/types/class.rs Outdated Show resolved Hide resolved

get(): More precise return type for unknown keys without default

4208fb4

AlexWaygood approved these changes Aug 29, 2025

View reviewed changes

sharkdp merged commit 5a608f7 into astral-sh:main Aug 29, 2025
38 checks passed

		# error: [invalid-argument-type] "Cannot pop required field 'name' from TypedDict `Person`"
		reveal_type(p.pop("name")) # revealed: Unknown

[ty] typecheck dict methods for TypedDict #19874

[ty] typecheck dict methods for TypedDict #19874

Uh oh!

Conversation

PrettyWood commented Aug 12, 2025 • edited by sharkdp Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

github-actions bot commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Diagnostic diff on typing conformance tests

Uh oh!

github-actions bot commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

github-actions bot commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ecosystem-analyzer results

Uh oh!

sharkdp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sharkdp commented Aug 29, 2025

Uh oh!

Uh oh!

dcreager left a comment

Choose a reason for hiding this comment

Uh oh!

dcreager Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

sharkdp Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sharkdp commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sharkdp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AlexWaygood Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

sharkdp Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[ty] typecheck dict methods for `TypedDict` #19874

[ty] typecheck dict methods for `TypedDict` #19874

PrettyWood commented Aug 12, 2025 •

edited by sharkdp

Loading

github-actions bot commented Aug 25, 2025 •

edited

Loading

github-actions bot commented Aug 25, 2025 •

edited

Loading

`mypy_primer` results

github-actions bot commented Aug 25, 2025 •

edited

Loading

`ecosystem-analyzer` results

sharkdp commented Aug 29, 2025 •

edited

Loading