Type stubs for single-file top-level modules

[not-my-profile]

PEP 561 currently states the following:

Package maintainers who wish to support type checking of their code MUST add a marker file named py.typed to their package supporting typing.

I consider this requirement to be problematic for Python libraries that are written in another programing language and distributed as compiled .so files. PEP 561 currently does not provide a way to mark .so files residing directly in site-packages/ to be typed, resulting in typed shared libraries needing to introduce an intermediary __init__.py file such as the following:

from ._native import *

__doc__ = _native.__doc__
if hasattr(_native, "__all__"):
    __all__ = _native.__all__

While this works for static type checkers I think this is obviously suboptimal because it has several undesired side-effects. Let's take the following as an example:

site-packages
├── my_project
│   ├── __init__.py
│   ├── _native.cpython-36m-x86_64-linux-gnu.so
│   └── py.typed

The unintended side-effects are:

1. You can import my_project._native.
2. _native shows up in the documentation generated by pydoc. E.g. under PACKAGE CONTENTS for the documentation of my_project and invoking e.g. help(my_project.foobar) will tell you that foobar resides in the module my_project._native.
3. my_project.__file__ now is the __init__.py file instead of the .so file, potentially misleading developers into thinking the package is implemented in Python

So I really think PEP 561 should be amended to provide some way of marking single-file packages as "typed" without having to resort to hacks such as defining an intermediary __init__.py since that introduces a bunch of undesired side-effects that have the potential to confuse API users.

What do you think about this?

[JelleZijlstra]

I agree this is suboptimal, and I'd support lifting the restriction if we can come up with a good way to do it. Maybe @ethanhs has some insights into why we didn't provide a way to type single-module packages at the time.

The first obvious solution perhaps would be to put something in the package's dist-info directory, e.g. a new key in the METADATA file. But the problem with that would be that type checkers can't reliably go from the name of the installed module to the dist-info directory, because the names may not match.

[not-my-profile]

A very simple solution could be to create a .typed marker in the same directory by appending .typed to the filename of the shared library so for example site-packages/myproject.cpython-36m-x86_64-linux-gnu.so could be marked as typed by creating the file site-packages/myproject.cpython-36m-x86_64-linux-gnu.so.typed.

I am unfamiliar with the process of amending a PEP, are there other places where I should announce this discussion? PEP 561 has been marked as "Final", does this mean that introducing such an update would require a new PEP?

[JelleZijlstra]

This would technically require a new PEP, yes. However, it can be short.

[hauntsaninja]

I think just {module_name}.typed would work and be easier for type checkers to search for

[not-my-profile]

Good point, Shantanu! Oh ... I just realized something ... static type checkers could just as well check for {module_name}.pyi within site-packages/ ... no need for a separate marker file at all.

Ok thanks Jelle, I'm working on a PEP draft for this right now.

[hauntsaninja]

I think .pyi as a marker file would only work for single module extension modules, not for single module pure Python files with inline types.

[not-my-profile]

Right I don't really consider having to change foo.py to foo/__init__.py to mark it as typed with inline types to be problematic because:

• documentation generators know how to deal with __init__.py files since they are so prevalent (e.g. pydoc doesn't list __init__ under PACKAGE CONTENTS and help displays in module foo instead of in module foo.__init__).
• foo.__file__ directly points to the source code

So the drawbacks I described that exist for shared libraries pretty much don't apply to pure Python modules.

I don't think we should introduce yet another type of .typed marker file since that is bound to result in confusion. What are the differences between foo.typed and py.typed? You can put partial\n in py.typed but not in foo.typed (since the stubs of single-file modules cannot be partial) so we would have two different types of marker files with the same extension, which is just confusing. Even worse there could be a package named py (in fact there is one on PyPI), so site-packages/py.typed would have different semantics than site-packages/foo/py.typed ... which again is confusing. I guess we could deal with that by introducing a .typed-module extension but this again isn't as clean since py.typed already exists and is named py.typed instead of py.typed-package.

I think we should rather go with the intuitive solution of putting a .pyi file in site-packages.

[erictraut]

I think we should rather go with the intuitive solution of putting a .pyi file in site-packages.

I agree this is the most straightforward approach for single-file binary (compiled) packages. Pyright already supports a ".pyi" file in site-packages if it's present. It looks like mypy doesn't handle this currently, but I'm guessing it would be a simple change.

[not-my-profile]

I wrote a draft PEP "Type stubs for single-file top-level modules". Feedback is very much welcome :)

(This is my first attempt at writing a PEP.)

[JelleZijlstra]

• I remember seeing some grumbling about needing py.typed even for single packages, but can't find it now. Maybe it was a package that wanted to remain distributed as a single .py file and therefore wanted typeshed stubs?
• Relevant related issue: Clarify the method of distribution for stub files with extension only packages. #1061
• The PEP should clarify where in the resolution order defined in PEP 561 (https://peps.python.org/pep-0561/#type-checker-module-resolution-order) the stub should go. (Presumably under # 4.) What happens if there is both a foo.pyi and a foo/ directory with a py.typed in it?
• You'll need a PEP sponsor; I'm happy to do it, but @hauntsaninja or @AlexWaygood are also eligible and I have several PEPs in flight already.

[AlexWaygood]

I'm supportive of the idea and I'm happy to co-sponsor :)

However, I wouldn't want to be the sole sponsor — I haven't written or sponsored a PEP before, so I'm not 100% sure of the exact process. I also feel like I'm pretty close to full capacity on my open-source commitments at the moment.

[AlexWaygood]

• I remember seeing some grumbling about needing py.typed even for single packages, but can't find it now.

I've seen this in a few places; I'll also see if I can dig up some references.