Experiment: removing setuptools._distutils

[Avasam]

Edit: I meant to open this against my own fork, but oh well, it's here now. I don't mind the visibility.
Still waiting on the following upstream changes to be published with setuptools:

• Mark distutils as py.typed pypa/distutils#338
• Explicit re-exports of submodules in distutils.command.__all__ pypa/distutils#345

Now that setuptools._distutils is as typed as typeshed is, this PR explores removing setuptools._distutils by using a partial setuptools stub.

Looks like from the context of our tests, mypy doesn't understand that the setuptools stub is partial and should reference the runtime setuptools._distutils. So I had to "teach" the mypy test about it by using [mypy-tests] section in METADATA.toml, and bring that feature to the stubtest and regression tests as well.

tests/* changes are easier to review by hiding whitespace changes, and would be done in a separate PR if accepted.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

setuptools (https://github.com/pypa/setuptools)
+ setuptools/discovery.py:404: error: Incompatible types in assignment (expression has type "list[str]", variable has type "None")  [assignment]
+ setuptools/discovery.py:426: error: Incompatible types in assignment (expression has type "dict[str, str]", variable has type "None")  [assignment]
+ setuptools/discovery.py:427: error: Incompatible types in assignment (expression has type "list[str]", variable has type "None")  [assignment]
+ setuptools/discovery.py:428: error: Incompatible types in assignment (expression has type "list[str]", variable has type "None")  [assignment]
+ setuptools/discovery.py:447: error: Incompatible types in assignment (expression has type "list[str]", variable has type "None")  [assignment]
+ setuptools/discovery.py:448: error: Argument 1 to "remove_stubs" has incompatible type "None"; expected "list[str]"  [arg-type]
+ setuptools/discovery.py:454: error: Incompatible types in assignment (expression has type "list[str]", variable has type "None")  [assignment]
+ setuptools/discovery.py:456: error: Argument 1 to "_ensure_no_accidental_inclusion" of "ConfigDiscovery" has incompatible type "None"; expected "list[str]"  [arg-type]
+ setuptools/logging.py:35: error: Incompatible types in assignment (expression has type Module, variable has type "Logger")  [assignment]
+ setuptools/config/setupcfg.py:178: error: Incompatible types in assignment (expression has type "dict[str, str]", variable has type "None")  [assignment]
+ setuptools/config/_apply_pyprojecttoml.py:269: error: Need type annotation for "existing" (hint: "existing: list[<type>] = ...")  [var-annotated]
+ setuptools/config/pyprojecttoml.py:428: error: Incompatible types in assignment (expression has type "dict[str, str]", variable has type "None")  [assignment]
+ setuptools/command/bdist_egg.py:133: error: Incompatible types in assignment (expression has type "list[Never]", variable has type "None")  [assignment]
+ setuptools/command/egg_info.py:239: error: No overload variant of "get" of "dict" matches argument types "str", "str"  [call-overload]
+ setuptools/command/egg_info.py:239: note: Possible overload variants:
+ setuptools/command/egg_info.py:239: note:     def get(self, Never, None = ..., /) -> None
+ setuptools/command/egg_info.py:239: note:     def get(self, Never, Never, /) -> Never
+ setuptools/command/egg_info.py:239: note:     def [_T] get(self, Never, _T, /) -> _T
+ setuptools/command/build_py.py:64: error: No overload variant of "copy_file" of "Command" matches argument types "str | PathLike[str]", "StrPathT", "bool", "bool", "str | None", "object"  [call-overload]
+ setuptools/command/build_py.py:64: note: Possible overload variants:
+ setuptools/command/build_py.py:64: note:     def [_StrPathT: str | PathLike[str]] copy_file(self, infile: str | PathLike[str], outfile: _StrPathT, preserve_mode: bool = ..., preserve_times: bool = ..., link: str | None = ..., level: int = ...) -> tuple[_StrPathT | str, bool]
+ setuptools/command/build_py.py:64: note:     def [_BytesPathT: bytes | PathLike[bytes]] copy_file(self, infile: bytes | PathLike[bytes], outfile: _BytesPathT, preserve_mode: bool = ..., preserve_times: bool = ..., link: str | None = ..., level: int = ...) -> tuple[_BytesPathT | bytes, bool]
+ setuptools/command/build_py.py:179: error: Need type annotation for "package"  [var-annotated]
- setuptools/command/build_clib.py:82: error: Value of type variable "_SourcesT" of "newer_pairwise_group" cannot be "list[Any]"  [type-var]
+ setuptools/command/editable_wheel.py:126: error: Need type annotation for "package_dir" (hint: "package_dir: dict[<type>, <type>] = ...")  [var-annotated]
+ setuptools/command/editable_wheel.py:209: error: Incompatible types in assignment (expression has type "str", variable has type "None")  [assignment]
+ setuptools/command/editable_wheel.py:210: error: Incompatible types in assignment (expression has type "str", variable has type "None")  [assignment]
+ setuptools/command/editable_wheel.py:211: error: Incompatible types in assignment (expression has type "str", variable has type "None")  [assignment]
+ setuptools/command/editable_wheel.py:212: error: Incompatible types in assignment (expression has type "str", variable has type "None")  [assignment]
+ setuptools/command/editable_wheel.py:506: error: Need type annotation for "package_dir" (hint: "package_dir: dict[<type>, <type>] = ...")  [var-annotated]
+ setuptools/command/editable_wheel.py:660: error: Need type annotation for "py_modules" (hint: "py_modules: list[<type>] = ...")  [var-annotated]
+ setuptools/command/editable_wheel.py:665: error: Need type annotation for "ext_modules" (hint: "ext_modules: list[<type>] = ...")  [var-annotated]
+ setuptools/command/editable_wheel.py:674: error: Need type annotation for "py_modules" (hint: "py_modules: list[<type>] = ...")  [var-annotated]
+ setuptools/command/editable_wheel.py:678: error: Need type annotation for "ext_modules" (hint: "ext_modules: list[<type>] = ...")  [var-annotated]

pip (https://github.com/pypa/pip)
+ src/pip/_internal/locations/_distutils.py:20: error: Module "distutils.cmd" has no attribute "Command"  [attr-defined]
- src/pip/_internal/locations/_distutils.py:67: error: Incompatible types in assignment (expression has type "Command", variable has type "install")  [assignment]
+ src/pip/_internal/locations/_distutils.py:22: error: Module "distutils.command.install" has no attribute "install"  [attr-defined]
+ src/pip/_internal/locations/_distutils.py:23: error: Module "distutils.sysconfig" has no attribute "get_python_lib"  [attr-defined]
+ src/pip/_internal/locations/_distutils.py:48: error: Module "distutils.dist" has no attribute "Distribution"  [attr-defined]
+ src/pip/_internal/locations/__init__.py:132: error: Module "distutils.command.install" has no attribute "install"  [attr-defined]
+ src/pip/_internal/locations/__init__.py:133: error: Module "distutils.dist" has no attribute "Distribution"  [attr-defined]

pybind11 (https://github.com/pybind/pybind11)
+ pybind11/setup_helpers.py:69: error: Unused "type: ignore" comment  [unused-ignore]
+ pybind11/setup_helpers.py:69: error: Module "distutils.command.build_ext" has no attribute "build_ext"  [attr-defined]
+ pybind11/setup_helpers.py:69: note: Error code "attr-defined" not covered by "type: ignore" comment
+ pybind11/setup_helpers.py:69: error: Name "_build_ext" already defined (possibly by an import)  [no-redef]
+ pybind11/setup_helpers.py:69: note: Error code "no-redef" not covered by "type: ignore" comment
+ pybind11/setup_helpers.py:72: error: Unused "type: ignore" comment  [unused-ignore]
+ pybind11/setup_helpers.py:72: error: Module "distutils.extension" has no attribute "Extension"  [attr-defined]
+ pybind11/setup_helpers.py:72: note: Error code "attr-defined" not covered by "type: ignore" comment
+ pybind11/setup_helpers.py:72: error: Name "_Extension" already defined (possibly by an import)  [no-redef]
+ pybind11/setup_helpers.py:72: note: Error code "no-redef" not covered by "type: ignore" comment
+ pybind11/setup_helpers.py:243: error: Module has no attribute "CompileError"  [attr-defined]
+ pybind11/setup_helpers.py:440: error: Unused "type: ignore" comment  [unused-ignore]
+ pybind11/setup_helpers.py:443: error: Unused "type: ignore" comment  [unused-ignore]
+ pybind11/setup_helpers.py:459: error: Unused "type: ignore" comment  [unused-ignore]
+ pybind11/setup_helpers.py:492: error: Unused "type: ignore" comment  [unused-ignore]
+ pybind11/setup_helpers.py:500: error: Unused "type: ignore" comment  [unused-ignore]

dd-trace-py (https://github.com/DataDog/dd-trace-py)
- ddtrace/sourcecode/setuptools_auto.py:11: error: Unused "type: ignore" comment  [unused-ignore]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/run-paasta-api-in-dev-mode.py:3: error: Module "distutils.dir_util" has no attribute "copy_tree"  [attr-defined]

aioredis (https://github.com/aio-libs/aioredis)
+ aioredis/connection.py:11: error: Module "distutils.version" has no attribute "StrictVersion"  [attr-defined]

mypy (https://github.com/python/mypy)
+ mypyc/build.py:55: error: Module "distutils.core" has no attribute "Extension"  [attr-defined]
+ mypyc/build.py:60: error: Type alias target becomes "Union[Extension, Any]" due to an unfollowed import  [no-any-unimported]
+ mypyc/build.py:78: error: Module has no attribute "Extension"  [attr-defined]

comtypes (https://github.com/enthought/comtypes)
- comtypes/test/setup.py:3: error: Cannot find implementation or library stub for module named "distutils.core"  [import-not-found]
+ comtypes/test/setup.py:3: error: Module "distutils.core" has no attribute "setup"  [attr-defined]

[Avasam]

The primer results for setuptools are either true positives (see pypa/setuptools#4861) or require updates in pypa/distutils (see pypa/distutils#357).

Other repos haven't been investigated yet.