improve type hint for python runtime and generated files?

[renzhentaxi]

I am currently using pyright as my LSP. When I try to use the python runtime and the generated parser/lexer/visitor... files for python3, I got a lot of syntax issues.

For example, the generated files prefer to use wild card imports which pyright doesn't seem to understand.:

from antlr4 import *

Possible Solution:
It looks like we are supposed to use a syntax like this in __init__.py

from antlr4.Token import Token as Token

instead of

from antlr4.Token import Token

we will also need to create a 'py.typed' marker file in the directory

Another issue is that the runtime uses empty classes as "interface".

Here are some of these "interfaces" in antlr4/tree/Tree.py:

class Tree(object):
   pass

class SyntaxTree(Tree):
   pass

class ParseTree(SyntaxTree):
   pass

class RuleNode(ParseTree):
   pass

class TerminalNode(ParseTree):
   pass

class ErrorNode(TerminalNode):
   pass

Because all of them are essentially blank/empty classes, It is impossible for a user or IDE to know what methods/properties are available on them.

Possible Solution:

We can either use stub files (.pyi) or Protocol (python 3.8)

Another issue is with 'forward declaration' which is not really supported by python.
In antlr4/RuleContext.py
We have:

# need forward declarations
RuleContext = None
Parser = None
...
...
    def toStringTree(self, ruleNames:list=None, recog:Parser=None):

Which essentially just means recog:None = None to the type system.

We are probably using Parser because of a circular dependency. E.g. RuleContext cann't import the actual Parser definition because the file that Parser is in is importing RuleContext.py. However, I think it might be better to just change the signature to recog = None (essentially recog:Any = None). Since we are not actually getting any benefits by typing it to 'None'

Possible solution:

we can use from __future__ import annotations (for 3.7+ I think) or use string.
e.g.

class Foo:
    def return_this(self) -> 'Foo':
         pass

Currently, we are dealing with textio via this line of code

which pyright doesn't understand.

However, it seems to understand it if we change it to:

[ericvergnaud]

Hi,
thanks for this.
Since AFAIK the Python 3 runtime and generated Lexer/Parser/Listener/Visitor have no syntax issues, this smells like a pyright issue.
Not saying there is no room for improvement in type hints but I would be very cautious with any PR, since the behaviour is largely tool dependent (as an example some tools are very happy with forward declarations)

[renzhentaxi]

Yep. I definitely agree that some of the issues are pyright oriented (e.g. I think mypy has no issue with wild card import)

However, I still think it would be really nice if the "interfaces" would actually the methods defined (with the proper signature and blank body), instead of just being a blank class. I tried to use different LSP for python (and different editors nvim/vscode) and none of them were able to tell me what methods are available on a TerminalNode (where as java implementations were able to).

Obviously this is because of the fact that python does not have "interfaces" but maybe we can move the Blank interfaces into a separate file (with no code changes besides from adjusting the import statements) and then create a type stub that contains the method definition for those interfaces (.pyi file with the same name). This way we would introduce no new runtime behavior while improving the python dev experience.

[ericvergnaud]

I am not hostile to the idea of enriching the existing 'abstract' classes aka 'interfaces' such that they expose methods. If you'll submit a PR (for both Python 2 and 3) I'll consider it

[Beakerboy]

interface stubs for mypy may soon be included at typeshed.