`libmagic` issues on macOS

[vpetersson]

This issue mentioned for Windows, but it is slightly different. On macOS, there's no such package as python-magic-bin.

$ pip install python-magic-bin
ERROR: Could not find a version that satisfies the requirement python-magic-bin (from versions: none)
ERROR: No matching distribution found for python-magic-bin

Full error message:

$ sbom4python
Traceback (most recent call last):
  File "/Users/mvip/tmp/sbom-lab/venv/bin/sbom4python", line 5, in <module>
    from sbom4python.cli import main
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/sbom4python/cli.py", line 14, in <module>
    from sbom4python.scanner import SBOMScanner
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/sbom4python/scanner.py", line 15, in <module>
    from sbom4files.filescanner import FileScanner
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/sbom4files/filescanner.py", line 7, in <module>
    import magic
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/magic/__init__.py", line 209, in <module>
    libmagic = loader.load_lib()
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/magic/loader.py", line 49, in load_lib
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

More details:

$ pip freeze
certifi==2024.7.4
charset-normalizer==3.3.2
defusedxml==0.7.1
idna==3.7
lib4package==0.2.0
lib4sbom==0.7.2
python-magic==0.4.27
PyYAML==6.0.1
requests==2.32.3
sbom2dot==0.3.0
sbom4files==0.4.2
sbom4python==0.11.0
semantic-version==2.10.0
urllib3==2.2.2

$ python --version
Python 3.9.19

[vpetersson]

I can confirm that I'm getting the same thing on Ubuntu:

$ sbom4python
Traceback (most recent call last):
  File "/code/venv2/bin/sbom4python", line 5, in <module>
    from sbom4python.cli import main
  File "/code/venv2/lib/python3.10/site-packages/sbom4python/cli.py", line 14, in <module>
    from sbom4python.scanner import SBOMScanner
  File "/code/venv2/lib/python3.10/site-packages/sbom4python/scanner.py", line 15, in <module>
    from sbom4files.filescanner import FileScanner
  File "/code/venv2/lib/python3.10/site-packages/sbom4files/filescanner.py", line 7, in <module>
    import magic
  File "/code/venv2/lib/python3.10/site-packages/magic/__init__.py", line 209, in <module>
    libmagic = loader.load_lib()
  File "/code/venv2/lib/python3.10/site-packages/magic/loader.py", line 49, in load_lib
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

[anthonyharrison]

Just installed sbom4python in a clean Python virtual machine on MacOS and it works.

Looking at the install logs python-magic (version 0.4.27) is installed but not python-magic-bin. python-magic-bin is installed via sbom4files. The requirements.txt file only installed python-magic-bin for Win32 platforms.

I am using Python version 3.11.1.

Can you try it in a clean python virtual machine?

[vpetersson]

Yeah this was in a clean virtual environment on both. One clean Docker Ubuntu container, and one on macOS.

$ virtualenv -p python3 venv
/opt/homebrew/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
created virtual environment CPython3.9.19.final.0-64 in 215ms
  creator CPython3Posix(dest=/Users/mvip/tmp/sbom-lab/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mvip/Library/Application Support/virtualenv)
    added seed packages: pip==24.1.2, setuptools==70.2.0, wheel==0.43.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$ source venv/bin/activate

$ pip freeze

$ pip install sbom4python
Collecting sbom4python
  Using cached sbom4python-0.11.0-py2.py3-none-any.whl.metadata (8.0 kB)
Collecting lib4package (from sbom4python)
  Using cached lib4package-0.2.0-py2.py3-none-any.whl.metadata (6.3 kB)
Collecting lib4sbom>=0.7.2 (from sbom4python)
  Using cached lib4sbom-0.7.2-py2.py3-none-any.whl.metadata (42 kB)
Collecting sbom2dot>=0.3.0 (from sbom4python)
  Using cached sbom2dot-0.3.0-py2.py3-none-any.whl.metadata (7.9 kB)
Collecting sbom4files>=0.4.0 (from sbom4python)
  Using cached sbom4files-0.4.2-py2.py3-none-any.whl.metadata (6.1 kB)
Collecting pyyaml>=5.4 (from lib4sbom>=0.7.2->sbom4python)
  Using cached PyYAML-6.0.1-cp39-cp39-macosx_11_0_arm64.whl.metadata (2.1 kB)
Collecting semantic-version (from lib4sbom>=0.7.2->sbom4python)
  Using cached semantic_version-2.10.0-py2.py3-none-any.whl.metadata (9.7 kB)
Collecting defusedxml (from lib4sbom>=0.7.2->sbom4python)
  Using cached defusedxml-0.7.1-py2.py3-none-any.whl.metadata (32 kB)
Collecting python-magic (from sbom4files>=0.4.0->sbom4python)
  Using cached python_magic-0.4.27-py2.py3-none-any.whl.metadata (5.8 kB)
Collecting requests (from lib4package->sbom4python)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests->lib4package->sbom4python)
  Using cached charset_normalizer-3.3.2-cp39-cp39-macosx_11_0_arm64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->lib4package->sbom4python)
  Using cached idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests->lib4package->sbom4python)
  Using cached urllib3-2.2.2-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests->lib4package->sbom4python)
  Using cached certifi-2024.7.4-py3-none-any.whl.metadata (2.2 kB)
Using cached sbom4python-0.11.0-py2.py3-none-any.whl (42 kB)
Using cached lib4sbom-0.7.2-py2.py3-none-any.whl (2.1 MB)
Using cached sbom2dot-0.3.0-py2.py3-none-any.whl (6.3 kB)
Using cached sbom4files-0.4.2-py2.py3-none-any.whl (12 kB)
Using cached lib4package-0.2.0-py2.py3-none-any.whl (9.5 kB)
Using cached PyYAML-6.0.1-cp39-cp39-macosx_11_0_arm64.whl (174 kB)
Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Using cached python_magic-0.4.27-py2.py3-none-any.whl (13 kB)
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Using cached certifi-2024.7.4-py3-none-any.whl (162 kB)
Using cached charset_normalizer-3.3.2-cp39-cp39-macosx_11_0_arm64.whl (120 kB)
Using cached idna-3.7-py3-none-any.whl (66 kB)
Using cached urllib3-2.2.2-py3-none-any.whl (121 kB)
Installing collected packages: urllib3, semantic-version, pyyaml, python-magic, idna, defusedxml, charset-normalizer, certifi, requests, lib4sbom, sbom4files, sbom2dot, lib4package, sbom4python
Successfully installed certifi-2024.7.4 charset-normalizer-3.3.2 defusedxml-0.7.1 idna-3.7 lib4package-0.2.0 lib4sbom-0.7.2 python-magic-0.4.27 pyyaml-6.0.1 requests-2.32.3 sbom2dot-0.3.0 sbom4files-0.4.2 sbom4python-0.11.0 semantic-version-2.10.0 urllib3-2.2.2

[notice] A new release of pip is available: 24.1.2 -> 24.2
[notice] To update, run: pip install --upgrade pip

$ sbom4python
Traceback (most recent call last):
  File "/Users/mvip/tmp/sbom-lab/venv/bin/sbom4python", line 5, in <module>
    from sbom4python.cli import main
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/sbom4python/cli.py", line 14, in <module>
    from sbom4python.scanner import SBOMScanner
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/sbom4python/scanner.py", line 15, in <module>
    from sbom4files.filescanner import FileScanner
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/sbom4files/filescanner.py", line 7, in <module>
    import magic
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/magic/__init__.py", line 209, in <module>
    libmagic = loader.load_lib()
  File "/Users/mvip/tmp/sbom-lab/venv/lib/python3.9/site-packages/magic/loader.py", line 49, in load_lib
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

[vpetersson]

And on ubuntu:

> $ docker run --rm -ti -v $(pwd):/code ubuntu bash
root@3cba3e6f7c39:/# apt -qq update && apt install -qqy python3-pip python3-virtualenv
25 packages can be upgraded. Run 'apt list --upgradable' to see them.
[snip]
done.
root@3cba3e6f7c39:/# cd /tmp/
root@3cba3e6f7c39:/tmp# ls
root@3cba3e6f7c39:/tmp# virtualenv venv
created virtual environment CPython3.10.12.final.0-64 in 129ms
  creator CPython3Posix(dest=/tmp/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.0.2, setuptools==59.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@3cba3e6f7c39:/tmp# source venv/bin/activate
(venv) root@3cba3e6f7c39:/tmp# pip install sbom4python
Collecting sbom4python
  Downloading sbom4python-0.11.0-py2.py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.3/42.3 KB 1.0 MB/s eta 0:00:00
Collecting lib4package
  Downloading lib4package-0.2.0-py2.py3-none-any.whl (9.5 kB)
Collecting sbom2dot>=0.3.0
  Downloading sbom2dot-0.3.0-py2.py3-none-any.whl (6.3 kB)
Collecting lib4sbom>=0.7.2
  Downloading lib4sbom-0.7.2-py2.py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 9.1 MB/s eta 0:00:00
Collecting sbom4files>=0.4.0
  Downloading sbom4files-0.4.2-py2.py3-none-any.whl (12 kB)
Collecting semantic-version
  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting defusedxml
  Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting pyyaml>=5.4
  Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (677 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 677.0/677.0 KB 5.7 MB/s eta 0:00:00
Collecting python-magic
  Downloading python_magic-0.4.27-py2.py3-none-any.whl (13 kB)
Collecting requests
  Downloading requests-2.32.3-py3-none-any.whl (64 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 KB 12.5 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.2/138.2 KB 11.1 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2024.7.4-py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.0/163.0 KB 10.1 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.2.2-py3-none-any.whl (121 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.4/121.4 KB 10.4 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.7-py3-none-any.whl (66 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.8/66.8 KB 13.9 MB/s eta 0:00:00
Installing collected packages: urllib3, semantic-version, pyyaml, python-magic, idna, defusedxml, charset-normalizer, certifi, requests, lib4sbom, sbom4files, sbom2dot, lib4package, sbom4python
Successfully installed certifi-2024.7.4 charset-normalizer-3.3.2 defusedxml-0.7.1 idna-3.7 lib4package-0.2.0 lib4sbom-0.7.2 python-magic-0.4.27 pyyaml-6.0.1 requests-2.32.3 sbom2dot-0.3.0 sbom4files-0.4.2 sbom4python-0.11.0 semantic-version-2.10.0 urllib3-2.2.2
(venv) root@3cba3e6f7c39:/tmp# pip freeze
certifi==2024.7.4
charset-normalizer==3.3.2
defusedxml==0.7.1
idna==3.7
lib4package==0.2.0
lib4sbom==0.7.2
python-magic==0.4.27
PyYAML==6.0.1
requests==2.32.3
sbom2dot==0.3.0
sbom4files==0.4.2
sbom4python==0.11.0
semantic-version==2.10.0
urllib3==2.2.2
(venv) root@3cba3e6f7c39:/tmp# sbom4python
Traceback (most recent call last):
  File "/tmp/venv/bin/sbom4python", line 5, in <module>
    from sbom4python.cli import main
  File "/tmp/venv/lib/python3.10/site-packages/sbom4python/cli.py", line 14, in <module>
    from sbom4python.scanner import SBOMScanner
  File "/tmp/venv/lib/python3.10/site-packages/sbom4python/scanner.py", line 15, in <module>
    from sbom4files.filescanner import FileScanner
  File "/tmp/venv/lib/python3.10/site-packages/sbom4files/filescanner.py", line 7, in <module>
    import magic
  File "/tmp/venv/lib/python3.10/site-packages/magic/__init__.py", line 209, in <module>
    libmagic = loader.load_lib()
  File "/tmp/venv/lib/python3.10/site-packages/magic/loader.py", line 49, in load_lib
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

[anthonyharrison]

On MacoS, I have a version of libmagic in /opt/local/lib.

On my Debian system, I have it in /usr/lib/x86_64-linux-gnu

[vpetersson]

You're missing a package in the installation requirements:

• Ubuntu: apt install libmagic-dev
• macOS: brew install libmagic

With that sorted, it runs fine.

[anthonyharrison]

You're missing a package in the installation requirements:

• Ubuntu: apt install libmagic-dev
• macOS: brew install libmagic

With that sorted, it runs fine.

Thanks. I will update the documentation but will see if there is a more elegant way of solving this.