UI freeze after parse on SQL grammars with large inputs

[Endeavourl]

I'm trying to profile my SQL grammar on a large input with diverse statements, which used to go fine in previous versions some years ago.
I managed to consistently reproduce a UI freeze that seems to happen after parsing of the input is finished.
Looking at the CPU load, there's a ~15s multithreaded parser spike and then a single thread becomes pinned at 100% and UI freezes.
Considering this and #381 it seems that the issue is with displaying parsing/profiling results. And it happens even when disabling every option on the left of the preview panel (auto-update, parse-tree etc).

Repro:
Use example PostgreSQL grammar or my PostgreSQL grammar.
Test root rule (sql rule in my grammar).
Concat every example SQL into a single file, i used this command: find *.sql | xargs -I{} sh -c "cat {}; echo ''" > _concat.sql.
Use the new, ~2.9MB file as ANTLR Preview input.
Note UI freeze after a short delay (parse stage). UI doesn't unfreeze for at least 10 minutes, process has to be killed.

[bjansen]

Hi, thanks for reporting the problem and for the reproducible case. Not sure how to handle this one. There might be ways to improve the situation, like parsing in a (possibly cancellable) background thread instead of the UI thread.
Have you tried parsing this rather huge file with grun directly? Depending on how the grammar is written, if the parsing itself is slow there's not much we can do to make it faster. Avoiding UI freezes should be possible, but there's no guarantee that the background thread won't still take 10min to process the file.
Also I'm not sure how the IDE will react when it has to render the parsing results in the preview window. This has to be done in the UI thread anyway.

[KvanTTT]

There might be ways to improve the situation, like parsing in a (possibly cancellable) background thread instead of the UI thread.

Unfortunately, currently ANTLR doesn't support cancellation: antlr/antlr4#2274

[Endeavourl]

The issue is not cancelability, obviously such large input is expected to produce some temporary UI stutter. Right now though it completely hangs in what seems to be an infinite loop. I let the process run for an hour and it didn't recover. It didn't run out of memory either.
In earlier (ca. 2019) plugin versions this kind of test worked ok, completing UI render in about a minute or so.

I tried running my grammar with grun method, it required removing java package lines from the @header sections, but otherwise worked fine. Parser took several seconds, and it even rendered the graph UI, obviously laggy but somewhat responsive.

To me this seems like an infinite loop somewhere in ANTLR Preview UI code, when it fills the parse result data.

[parrt]

I'm pretty sure that the tree visualization is the issue; it's either nonlinear or mildly so. I wonder if we could simply put the tree construction in a separate thread and kill it if it takes too long? I ran into this issue myself recently.

[bjansen]

@KvanTTT we could do something like override org.antlr.v4.runtime.ParserInterpreter#visitState and throw an exception when needed ;)

[Thracky]

Would like to add my +1 to this issue and note that I am seeing it on a grammar of similar complexity and a much smaller input, only about 310 KB.

[parrt]

hi. I totally agree. I'm going to try to solve this hopefully this weekend. The basic problem is that the tree layout mechanism is nonlinear and I stupidly build the tree picture using the GUI thread I think. Upon keystroke, I need to initiate thread worker to build the Picture but it must kill any Picture construction that is going on already. I don't think this has anything to do with parsing speed canceling the parse. @bjansen does that sound reasonable?

[parrt]

Hi. @Thracky @Endeavourl Can you guys try a snapshot of 1.19? antlr-intellij-plugin-v4-1.19-SNAPSHOT.zip

[Endeavourl]

Hangs exactly like the stable version for me.

[parrt]

Dang. i tried it on java grammar and big input. had no problem. Well, it's slow parsing like 8s but doesn't freeze. You sure it installed correctly?

[parrt]

I'm happy to get your grammar and input if public...ah i see the SQL grammars. i'll try it out. wow. ok, 76k lines? Hmm...not sure that's gonna ever work in interp mode rather than gen'd code executing.

[parrt]

Confirmed. 76k lines kills it. Hmm...should work...investigating.

Wow. It's the hierarchy view update that is super slow. And I don't think it's the initial view...it's the update for new input.

Nope. It's caused by the views but isn't the views. Might be memory pressure / thrashing? Still digging. Damn thing freezes after displaying tree. 5000 lines of SQL makes it jump 2G in RAM. Also I note it's using intel for the 2020.2.1. Trying 2021.3.3 now.