Support for type hints

[art049]

Hi,

I'm thinking about adding something to provide static type support. To be able to use umongo with mypy for example.

I think there is not really an easy way to provide this kind of feature since each DocumentImplementation method can have it's own signature.
Therefore, each implementation will be type annotated. For now, I'm unsure this will be enough for a mypy commpatibility.

I'd love to hear your thoughts about this :)

[lafrech]

Hi.

I'm totally open to using type hints here.

umongo 3.x (still beta) now uses marshmallow and provides type hints. Plus I'm about to drop Python 3.5 compatibility (#209) if that helps.

I can't comment about your question. I'm not familiar enough with type hints. But PRs implementing this are absolutely welcome.

[talwrii]

Useful information

Hey,

I don't have experience with this, but I do have a some interest, google, and I spent a few hours trying to write some similar code for pylint (pylint and mypy are quite similar really).

mypy supports custom plugins. I found this documentation difficult to follow, but source code is well documented with example use cases.

You probably would want to use get_base_class_hook which can modify the code structure (I'm hesistant to call it an ast without a little more understand) that mypy is analysing.

Django contains one of the most used ORMs so is a good place to look for sample code. There is a project to make mypy more compatible with mypy. Here is how it uses this method. This project is quite large.

Rambling into the void about macros, metaprogramming, parse transformers, linters and code duplication

I'm not sure if this sort of project makes me happy. It feels as if these plugins can contain a lot of code, which is reimplementing the magic that ORMs do at runtime at an AST level, and doing so in quite a fiddly way. And this code has to be kept in sync with another project, and might have complicated errors. I don't know what a better way is though :/ , ideally one would make the static analysis and metaclass magic run at the same time. To achieve this you could have your metaprogramming run on the AST and then the type checker run on this generated AST (preferrably with some mechanisms to work out which lines in the source code correspond to the new AST). This is how erlang and lisps metaprogramming works. You might actually be able to have a module parse and modify its own ast and load the modified version into sys.modules on import, and then one hopes, you could get the type checker to work with this modified version as well.

You lose something for going in this direction though. Firstly your metaprogramming code is less flexible because it doesn't get to access data at runtime, only types, and secondly you start having different types of code, different languages, and different types of execution. One of the wonderful thing about python is that there is only really one type of execution and model of execution to understand, where as in other languages different things execute in different ways with different rules, which can make things a whole lot more complicated.

Going in the other direction (getting informaion out of your metaprogrammer for your type checker), you could actually execute your metaprogramming code in python and then extract type information from this, and then use this in your type checking. This is obviously terrible. I remember reading about someone who was automatically generating mypy type annotations in this manner as a form of "consistency type validation". This is a bit of horrible fusion of run time and compile time.

Everything is terrible, glue, code duplication, process and elbow grease solves all problems!

[ThibaultLemaire]

Hi, I'm unfamiliar with umongo's code, but I am willing to start working on this.

For reference, here is the documentation on how to add type hints to a package. Considering the dynamic nature of uMongo (as far as I understand) I think I'll start by writing stubs (.pyi) files rather than trying to annotate existing code.

[lafrech]

Thanks for offering to help on this.

I don't understand the need for a stub file vs. inline typing info.

You may submit PRs with typing info. It doesn't have to be a single complete PR. Partial is better than nothing.

I won't be able to provide much help on this (not familiar + little time).

[ThibaultLemaire]

I don't understand the need for a stub file vs. inline typing info.

Mypy analyses files recursively. So when type checking some end-user code it will also analyse umongo's source files. If they don't typecheck, then the end-user will see a wall of errors from code they didn't even write.

So either you

1. fix all your codebase for mypy (which might be a good thing and might catch some edge-cases), or
2. you provide type stubs as a shield so mypy will stop there and not go into the intricacies of your code (which is also convenient when you have some meta-programming that changes the underlying types at run-time).

By choosing to go with 2. I can also spare myself the understanding of the whole umongo codebase and simply look at it from a user perspective.

(You can also mix the two approaches, and choosing to start with stubs doesn't mean we won't be able to switch to inline annotations later on. Which are better for maintenance since stubs have to be kept in sync with the actual codebase.)