CoLic throwing path error

[altsalt]

Hello, I am participating in the MSR Hackathon and have attempted to install Graal a few times to no avail. Most recently, I set up a clean Debian 11 VM to verify that it wasn't due to prior system configuration, incorrect permissions, etc. Alas, the same error is appearing:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from graal.backends.core.colic import CoLic
>>> repo_uri = 'https://github.com/chaoss/grimoirelab-graal'
>>> repo_dir = 'grimoirelab-graal'
>>> cl = CoLic(uri=repo_uri, git_path=repo_dir)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/graal-0.2.9-py3.9.egg/graal/backends/core/colic.py", line 79, in __init__
    if not GraalRepository.exists(exec_path):
  File "/usr/local/lib/python3.9/dist-packages/graal-0.2.9-py3.9.egg/graal/graal.py", line 411, in exists
    return os.path.exists(dest)
  File "/usr/lib/python3.9/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
>>> 

I have tried with both a blank directory and having pre-created the grimoirelab-graal folder (not that this should be a blocker, but looking for anything that might be...)

Any pointers or guidance would be very much appreciated. Thanks!

[altsalt]

Related issues:

• [scancode] Use scancode from python3 interface #64
• Cannot call (easily) the main scancode function from Python #1780

Also, while this issue is referencing the git path, it may be relevant.

[altsalt]

I've spent hours trying to get graal and scancode to play nice to no avail. I think this is largely due to my python path setup, but given that I have a fresh VM that only being used for this, there is likely an issue with the build instructions.

My latest attempted has graal being built into scancode-toolkit/bin using --prefix. This command is then successful:
(scancode-toolkit) $ graal colic https://github.com/chaoss/grimoirelab-graal --exec-path /home/salt/MSR-Magna_Carta/scancode-toolkit/bin/scancode --category code_license_scancode > ./scancode_test

However, as a script it continues to throw a ModuleNotFoundError. I've tried loading it directly via /home/salt/scancode-toolkit/bin/python3 test.py with or without the virtualenv active. I've also tried a variety of sys.path.append calls. I'm just not sure what I'm missing here....

The test script

#!/home/salt/MSR-Magna_Carta/scancode-toolkit/bin/python3

#import sys
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/bin')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/lib')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/python3.9/site-packages/')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/python3.9/site-packages/graal-0.2.9-py3.9.egg/graal/')

from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = 'grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir)

# fetch all commits
commits = [commit for commit in cl.fetch()]

with open("graal_test_commits", "w") as f:
  f.write(commits)

Some of the commands and errors which have gone by

(scancode-toolkit) $ graal colic https://github.com/chaoss/grimoirelab-graal --exec-path /home/salt/MSR-Magna_Carta/scancode-toolkit/etc/scripts/scancli.py  --category code_license_scancode_cli > ./scancode_cli_test
[2021-12-07 11:40:16,258] - Starting the quest for the Graal.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
[2021-12-07 11:40:16,592] - Error!: Command '['/home/salt/MSR-Magna_Carta/scancode-toolkit/configure']' returned non-zero exit status 1.
[2021-12-07 11:40:16,593] - Quest completed.

[valeriocos]

Hi @altsalt,

I followed the steps here https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executables, section ScanCode

cd /tmp
git clone https://github.com/nexB/scancode-toolkit.git
cd scancode-toolkit
git checkout -b test_scancli 96069fd84066c97549d54f66bd2fe8c7813c6b52
./scancode --help

pip install simplejson execnet

The execution from command line and python were OK

• command line

graal colic https://github.com/chaoss/grimoirelab-graal --git-path /tmp/test --exec-path /tmp/scancode-toolkit/etc/scripts/scancli.py --category code_license_scancode_cli

• python

from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = '/tmp/grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir, exec_path='/tmp/scancode-toolkit/etc/scripts/scancli.py')

# fetch all commits
commits = [commit for commit in cl.fetch(category='code_license_scancode_cli')] <--- this call may take a lot of time, you may want to save each commit in the file below.

with open("graal_test_commits", "w+") as f:
  f.write(commits)

Hope this helps

EDIT:

Also the category code_license_scancode seems to work fine

• command line

graal colic https://github.com/chaoss/grimoirelab-graal --git-path /tmp/test --exec-path /tmp/scancode-toolkit/scancode --category code_license_scancode

• python

from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = '/tmp/grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir, exec_path='/tmp/scancode-toolkit/scancode')

# fetch all commits
commits = []
for commit in cl.fetch(category='code_license_scancode'):
  commits.append(commit)
  print(commit)

with open("graal_test_commits", "w+") as f:
  f.write(commits)

[altsalt]

An error has been thrown in step 1. I'll install the missing package and keep pushing through, but it may point to part of the problem.

salt@debianclivm:/tmp/scancode-toolkit$ ./scancode --help
* Configuring ScanCode for first use...
/tmp/scancode-toolkit/configure: line 24: python2.7: command not found
./scancode: line 114: /tmp/scancode-toolkit/bin/scancode: No such file or directory

FWIW, I had been building ScanCode from the main branch rather than checking one out.