Error Parsing C++ Files for Code2Seq

[estiver-alvarez]

Hi sir, I want to take this occasion to express my admiration for your great job extremely useful for my Bachelor's thesis. 
Currently, I am trying to convert my dataset of about  100000 cpp files, but always I am getting this error.

Originally thought that the issue was related to the capacity of memory, but quickly this was dismissed because I work with instances of machines of up to 150G Ram, also I checked of limits set by the OS linux and these are far above of the  thread quantity created when the App crash. even thanks to the capacity of instances of the machines   with whom I have worked,  it's possible create a thread by each cpp file and the system would not have any problem of performance or configuration.

I have kept track of the logs JVM and with the help an analyzing tool. I found that over 99% thread is established in state-timed waiting and all these are asociated to MVStore.

may you help me with this issue.

Thank you very much for your attention

[estiver-alvarez]

this is the configuration that currently I'm applying.

[illided]

Hi! At the start astminer will tell you how many threads will be created (by default only 1 but you can change it in the config with numOfThreads option). Could you send us full console output and log.txt (it should be created in root of the project)?

[estiver-alvarez]

Hi! At the start astminer will tell you how many threads will be created (by default only 1 but you can change it in the config with numOfThreads option). Could you send us full console output and log.txt (it should be created in root of the project)?

of course this the file, thanks for you suport.
log.txt

[illided]

It's really strange. Astminer created only 1 thread, but some subcomponents created multiple threads. Could you also provide us with some of your's system info: OS, gcc compiler version, JDK version etc. I think anything can help :)

In the meantime you can run astminer from docker. For this run ./gradlew shadowJar and docker build -t voudy/astminer . in the root of the project. From now on cli script will run docker container instead of jar.

[estiver-alvarez]

It's really strange. Astminer created only 1 thread, but some subcomponents created multiple threads. Could you also provide us with some of your's system info: OS, gcc compiler version, JDK version etc. I think anything can help :)

In the meantime you can run astminer from docker. For this run ./gradlew shadowJar and docker build -t voudy/astminer . in the root of the project. From now on cli script will run docker container instead of jar.

@illided Thank you very much for your support. unfortunately still I have the issue,although this time the processing went so far than the previous tried, but likewise this couldn't make it at least 20%, the strangest it was that this time I executed it over cli and docker. the app is being run over an instance machine with 4 cpu and 132 gb ram, under linux(ubuntu 20.04 and debian 10) and with stable latest version of java 11. 

pdt when I executed the app under docker, I didn't know configure the parameters associated to heap in Java like -Xms -Xms for this reason I executed the app of the default way. 

I share you the last 100000 lines of log
log1.txt

[illided]

Ok, now it's super strange :)
Is it possible for you to share the data on which you run the astminer? If the dataset is very large at least a small part.

[illided]

Also, if you are in a real hurry, you can implement C ++ support through a tree sitter grammar. Parser that we are using for c++ right now is not the most stable.

We are planing to do this in the future, but PR is welcome.

[estiver-alvarez]

Ok, now it's super strange :) Is it possible for you to share the data on which you run the astminer? If the dataset is very large at least a small part.

@illided of course very kind of you :). here in the next link you will find the dataset that I'm trying to process.
https://drive.google.com/file/d/1CyNKBDYequb7izMAsk3wHj9A0HOWZsfc/view?usp=sharing

[pppyx]

It's really strange. Astminer created only 1 thread, but some subcomponents created multiple threads. Could you also provide us with some of your's system info: OS, gcc compiler version, JDK version etc. I think anything can help :)
In the meantime you can run astminer from docker. For this run ./gradlew shadowJar and docker build -t voudy/astminer . in the root of the project. From now on cli script will run docker container instead of jar.

@illided Thank you very much for your support. unfortunately still I have the issue,although this time the processing went so far than the previous tried, but likewise this couldn't make it at least 20%, the strangest it was that this time I executed it over cli and docker. the app is being run over an instance machine with 4 cpu and 132 gb ram, under linux(ubuntu 20.04 and debian 10) and with stable latest version of java 11. 

pdt when I executed the app under docker, I didn't know configure the parameters associated to heap in Java like -Xms -Xms for this reason I executed the app of the default way. 

I share you the last 100000 lines of log log1.txt

I'm now running into the same situation:OOM error when trying to create thread, and my dataset is also very large.

Hope you will find out the reason soon, thanks!

[illided]

Sorry, didn't had a time to respond.

As I suspected fuzzy parser creates multiple threads and can't close them properly. In fact it creates up to 6 threads for each file! This issue is fixed in new version of fuzzy parser, but unfortunately it's not compatible with astminer and we have no time right now to fully rewrite this part :(

You can try to experiment with the code, or, as i said earlier, try to implement support for c++ through tree-sitter. It shouldn't be that hard, as there a lot of examples of what you need to implement.

See astminer/src/main/kotlin/astminer/parse/treesitter/java/

[pppyx]

Sorry, didn't had a time to respond.

As I suspected fuzzy parser creates multiple threads and can't close them properly. In fact it creates up to 6 threads for each file! This issue is fixed in new version of fuzzy parser, but unfortunately it's not compatible with astminer and we have no time right now to fully rewrite this part :(

You can try to experiment with the code, or, as i said earlier, try to implement support for c++ through tree-sitter. It shouldn't be that hard, as there a lot of examples of what you need to implement.

See astminer/src/main/kotlin/astminer/parse/treesitter/java/

Hi, illided. Thank you for your prompt reply, I write a shell script to run astminer on a small batch of dataset each time and concat the path_context.c2s, I think this can be a quick solution for this problem temporarily.

[illided]

Sounds interesting! Could you share this solution here in case anyone will get the same problem?

[estiver-alvarez]

Sorry, didn't had a time to respond.
As I suspected fuzzy parser creates multiple threads and can't close them properly. In fact it creates up to 6 threads for each file! This issue is fixed in new version of fuzzy parser, but unfortunately it's not compatible with astminer and we have no time right now to fully rewrite this part :(
You can try to experiment with the code, or, as i said earlier, try to implement support for c++ through tree-sitter. It shouldn't be that hard, as there a lot of examples of what you need to implement.
See astminer/src/main/kotlin/astminer/parse/treesitter/java/

Hi, illided. Thank you for your prompt reply, I write a shell script to run astminer on a small batch of dataset each time and concat the path_context.c2s, I think this can be a quick solution for this problem temporarily.

please, may you share the script .