ParserATNSimulator debug crash on Python3 target

[sinopsysHK]

Running Grammars-v4/plsql generated parser and putting ParserATNSimulator in debug mode as bellow is crashing when running over simple SQL script:

SELECT * FROM TAB1 T1, TAB2 T2 WHERE T1.F1=T2.F1;

""" sample code """
import sys
from antlr4 import *
from antlr4.tree.Tree import TerminalNode
from myparser.PlSqlLexer import PlSqlLexer
from myparser.PlSqlParser import PlSqlParser
from myparser.PlSqlParserVisitor import PlSqlParserVisitor

class UpperFileStream(FileStream):
    def readDataFrom(self, fileName: str, encoding: str, errors: str = 'strict'):
        return super().readDataFrom(fileName, encoding, errors).upper()

ParserATNSimulator.debug = True
input = UpperFileStream(sys.argv[1])
input = FileStream(argv[1])
lexer = PlSqlLexer(input)
stream = CommonTokenStream(lexer)
parser = PlSqlParser(stream)
tree = parser.sql_script()_

with following stack trace:

  File "C:\Users\ericb_000\Documents\devs\langparser\src\myparser\PlSqlParser.py", line 12717, in unit_statement
    self.data_manipulation_language_statements()
  File "C:\Users\ericb_000\Documents\devs\langparser\src\myparser\PlSqlParser.py", line 63851, in data_manipulation_language_statements
    self.select_statement()
  File "C:\Users\ericb_000\Documents\devs\langparser\src\myparser\PlSqlParser.py", line 65257, in select_statement
    self.subquery()
  File "C:\Users\ericb_000\Documents\devs\langparser\src\myparser\PlSqlParser.py", line 65803, in subquery
    self.subquery_basic_elements()
  File "C:\Users\ericb_000\Documents\devs\langparser\src\myparser\PlSqlParser.py", line 65868, in subquery_basic_elements
    self.query_block()
  File "C:\Users\ericb_000\Documents\devs\langparser\src\myparser\PlSqlParser.py", line 66112, in query_block
    la_ = self._interp.adaptivePredict(self._input,1253,self._ctx)
  File "C:\ProgramData\Anaconda3\lib\site-packages\antlr4\atn\ParserATNSimulator.py", line 342, in adaptivePredict
    alt = self.execATN(dfa, s0, input, index, outerContext)
  File "C:\ProgramData\Anaconda3\lib\site-packages\antlr4\atn\ParserATNSimulator.py", line 442, in execATN
    alt = self.execATNWithFullContext(dfa, D, s0_closure, input, startIndex, outerContext)
  File "C:\ProgramData\Anaconda3\lib\site-packages\antlr4\atn\ParserATNSimulator.py", line 625, in execATNWithFullContext
    self.reportContextSensitivity(dfa, predictedAlt, reach, startIndex, input.index)
  File "C:\ProgramData\Anaconda3\lib\site-packages\antlr4\atn\ParserATNSimulator.py", line 1621, in reportContextSensitivity
    ", input=" + self.parser.getTokenStream().getText(interval))
  File "C:\ProgramData\Anaconda3\lib\site-packages\antlr4\BufferedTokenStream.py", line 283, in getText
    stop = interval[1]
IndexError: range object index out of range

I know that this is a limited issue but I wanted to use debug mode to investigate why this simple sql script takes more than a minute to parse in Python3 while it takes seconds with Java target.

[KvanTTT]

What version of ANTLR do you use?

[sinopsysHK]

Crashing on 4.7.1 release

[ericvergnaud]

Hi,
the conditional debug code hasn't been tested a lot I must admit.
This looks like a reminiscence of the tuple based implementation of Interval, instead of the xrange one (see IntervalSet.py)
Please feel free to adjust the code accordingly and submit a PR.

[ericvergnaud]

And back to the performance concern, as you probably know, pure python is 20-30 times slower than Java.
We observe roughly the same performance ratio in antlr.

[sinopsysHK]

Hi,
indeed I had a very short sight analysis (discovering ANTLR universe).
Looks like intervalSet hasn't been propagate every where !?
According to getText signature in BufferTokenStream.py used as Parser token stream:

# Get the text of all tokens in this buffer.#/
def getText(self, interval:tuple=None):
...

It is the same usage in TokenStreamRewriter.py

I apologize but I don't feel confident enough to streamline those last two.

[sinopsysHK]

Hello,

@ericvergnaud regarding performances, I would agree that python is 20-30 times slower than java target.

but with the simple and short sql script shared earlier, I have:
On Java:
Tree parsed in 1575 millisecs
On Python3:
tree parsed in 0:01:29.457739

This is then more like 60 times slower and gap is increasing with larger scripts (going up to 9min vs few secs)

[sinopsysHK]

Hello,

Hi,
the conditional debug code hasn't been tested a lot I must admit.
This looks like a reminiscence of the tuple based implementation of Interval, instead of the xrange one (see IntervalSet.py)
Please feel free to adjust the code accordingly and submit a PR.

Reading the various code in Python target I'm not clear what would be the correct change to apply.
I can see that python code try to stick to java one, but the Interval hasn't been implemented.
The other classes that supply a getText method upon an interval seems to rather use getText(startIndex, stopIndex).

@ericvergnaud : What would you advise ? moving BufferTokenStream and TokenStreamRewriter to the same signature than InputStream.getText ?

I can try to adjust and test accordingly...