(Django)(MAC Big Sur 11.0.1) ImportError: dlopen

[awildjosh]

Apologies in advance -- this is probably doubling up on #1200 and #1208, but I'm curious as to whether there's any workaround whatsoever (as I really would like to use postgresql with my project, but I'm too much of a newb to comprehend most of the issues I run into). I've tried pretty much every suggestion to be found through googling to no avail. The issue is as below:

• Mac Big Sur (OS 11.0.1)
• Python 3.9 (venv)

Installed pyscopg2 to the venv without any issues using:
pip install psycopg2

Meanwhile, Postgresql was successfully installed and the db I'd like to connect to is active (using pgadmin4). Django settings are configured accordingly.

However, when trying to access psycopg2 through the terminal, whether it be through:
python -c "import psycopg2" or python manage.py migrate, etc.

The below error occurs:
(venv) ...-MacBook-Air project % python -c "import psycopg2" Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa ImportError: dlopen(/Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): no suitable image found. Did find: /Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture /Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture

Any help/insight would be greatly appreciated. Or are all of us who jumped ship to M1 doomed until Apple stop breaking things? :)

[bschollnick]

Second this. Unable to use natively under M1. I am forced to use under Intel Translation / Rosetta 2.

Psycopg3 also does not compile natively...

[dvarrazzo]

Is there an M1 to test with?

[bschollnick]

I'm more than happy to volunteer my own time to test... I can either run whatever tests you need, or I can volunteer to allow you to remote in, etc.

[dvarrazzo]

I would appreciate that, thank you.

Most likely rebuilding packages using the newer pypa infrastructure would just solve the issue.

Could you please checkout https://github.com/psycopg/psycopg2-wheels/ and check if the scripts/build_macos.sh produces working packages?

Thank you very much.

Ref. pypa/pip#9138

[bschollnick]

As far as I can tell, with Rosetta 2, everything was fine (which seems correct, since I can install via PIP under Rosetta 2).

README.rst	appveyor.yml	data		psycopg2	scripts
benjamin@Nerv psycopg2-wheels-master % cd scripts
benjamin@Nerv scripts % build_libpq_macos.sh
configure: error: library 'crypto' is required for OpenSSL
benjamin@Nerv scripts % pip install crypto
Defaulting to user installation because normal site-packages is not writeable
Collecting crypto
  Downloading crypto-1.4.1-py2.py3-none-any.whl (18 kB)
Collecting Naked
  Downloading Naked-0.1.31-py2.py3-none-any.whl (590 kB)
     |████████████████████████████████| 590 kB 2.9 MB/s 
Requirement already satisfied: requests in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from Naked->crypto) (2.25.1)
Collecting pyyaml
  Using cached PyYAML-5.3.1-cp38-cp38-macosx_10_14_x86_64.whl
Requirement already satisfied: idna<3,>=2.5 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (1.26.2)
Requirement already satisfied: chardet<5,>=3.0.2 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (2020.12.5)
Collecting shellescape
  Downloading shellescape-3.8.1-py2.py3-none-any.whl (3.1 kB)
Installing collected packages: pyyaml, shellescape, Naked, crypto
  WARNING: The script naked is installed in '/Users/benjamin/Library/Python/3.8/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts crypto and decrypto are installed in '/Users/benjamin/Library/Python/3.8/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Naked-0.1.31 crypto-1.4.1 pyyaml-5.3.1 shellescape-3.8.1
benjamin@Nerv scripts % build_libpq_macos.sh
Makefile:14: ../../../src/Makefile.global: No such file or directory
Makefile:94: /src/Makefile.shlib: No such file or directory
make: *** No rule to make target `/src/Makefile.shlib'.  Stop.
benjamin@Nerv scripts % build_libpq_macos.sh
Makefile:14: ../../../src/Makefile.global: No such file or directory
Makefile:94: /src/Makefile.shlib: No such file or directory
make: *** No rule to make target `/src/Makefile.shlib'.  Stop.
benjamin@Nerv scripts % build_libpq_macos.sh
benjamin@Nerv scripts % ls      
build-manylinux.sh		build_manylinux1_x86_64.sh
build_libpq.sh			build_manylinux2014_aarch64.sh
build_libpq_macos.sh		build_sdist.sh
build_macos.sh			configure_postgres.sh
build_manylinux1_i686.sh	upload_packages.sh
benjamin@Nerv scripts % build_macos.sh
+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(binary-io.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(getprogname.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(lock.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-context.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-selinux.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(stat-time.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(threadlib.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(unistd.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(wctype-h.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(localtime-buffer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(binary-io.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(getprogname.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(lock.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-context.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-selinux.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(stat-time.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(threadlib.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(unistd.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(wctype-h.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(localtime-buffer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(fd-hook.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(localename-table.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sig-handler.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sys_socket.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(thread.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(fd-hook.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(localename-table.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sig-handler.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sys_socket.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(thread.o) has no symbols
+ make install
/Library/Developer/CommandLineTools/usr/bin/make  install-recursive
Making install in po
if test "sed" = "gettext-tools"; then \
	  ../build-aux/install-sh -c -d /usr/local/Cellar/gnu-sed/4.8/share/gettext/po; \
	  for file in Makefile.in.in remove-potcdate.sin quot.sed boldquot.sed en@quot.header en@boldquot.header insert-header.sin Rules-quot   Makevars.template; do \
	    /usr/bin/install -c -m 644 ./$file \
			    /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	  for file in Makevars; do \
	    rm -f /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	else \
	  : ; \
	fi
Making install in .
 build-aux/install-sh -c -d '/usr/local/Cellar/gnu-sed/4.8/bin'
mkdir: /usr/local/Cellar: Permission denied
mkdir: /usr/local/Cellar: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed/4.8: No such file or directory
make[3]: *** [install-binPROGRAMS] Error 1
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install] Error 2
benjamin@Nerv scripts % 

But at the end, it dies looking for Cellar? I don't know what that's about?

Trying next under Native M1.

[bschollnick]

First attempt:

(testing) benjamin@Nerv scripts % build_macos.sh      
+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(c-ctype.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(stripslash.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(threadlib.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(strerror-override.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(c-ctype.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(stripslash.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(threadlib.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(strerror-override.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: for architecture: x86_64 file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: for architecture: x86_64 file: lib/libsed.a(threadlib.o) has no symbols
make[2]: *** [lib/libsed.a] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

But that may be due to previously running the build under Rosetta 2 in that folder structure.
I nuked it, and re-unarchived it, which resulted in:

+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
+ make install
/Library/Developer/CommandLineTools/usr/bin/make  install-recursive
Making install in po
if test "sed" = "gettext-tools"; then \
	  ../build-aux/install-sh -c -d /usr/local/Cellar/gnu-sed/4.8/share/gettext/po; \
	  for file in Makefile.in.in remove-potcdate.sin quot.sed boldquot.sed en@quot.header en@boldquot.header insert-header.sin Rules-quot   Makevars.template; do \
	    /usr/bin/install -c -m 644 ./$file \
			    /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	  for file in Makevars; do \
	    rm -f /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
	  done; \
	else \
	  : ; \
	fi
Making install in .
 build-aux/install-sh -c -d '/usr/local/Cellar/gnu-sed/4.8/bin'
mkdir: /usr/local/Cellar: Permission denied
mkdir: /usr/local/Cellar: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed/4.8: No such file or directory
make[3]: *** [install-binPROGRAMS] Error 1
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install] Error 2

But a significantly faster built (Rosetta took about 2-3 minutes?, M1 about 20-30 seconds).

And ended with the seller / sed messages above.

[dvarrazzo]

make install should run as sudo I think. This script is designed to run in Travis with a superuser account.

[bschollnick]

I'm trying to make sense out of the build_macos script?
And I'm failing hard. There are a lot of moving parts here, and I'm not making headway on why any of it is there...

And the more I look through the build script, it's heavily personalized, and is requiring virtual environments that I don't have configured, and home-brew which is not yet working reliably under the native M1.

I couldn't get the scripts as supplied to work under either Intel Mode, or M1 Native mode.

Instead I just grabbed the psycopg2 dist from GitHub, and ran:

psycopg2 % python3 setup.py sdist bdist_wheel

Which created the Intel and m1 native based wheels. I'm unclear on what all the futzing around in the other scripts is suppose to accomplish?

I haven't had a chance to test these yet...

psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl.zip

psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_x86_64.whl.zip

[dvarrazzo]

The futzing is to create self-contained pacakges, containing the libpq. Without it, the wheels need some system libraries to run, and because on mac os there is no standard for installation, people complain that it's too difficult.

I'll try and re-run the packages build, maybe you can help me testing if the packages work alright.

[dvarrazzo]

@bschollnick Do packages available here work? https://upload.psycopg.org/psycopg2-2.8.7.dev0/ Thank you!

[bschollnick]

Works fine under Intel Translation, but fails under M1 Native.

(testing) Nerv ~ % pip install /Volumes/4TB_Drive/sorted_downloads/2021-01-11/psycopg2_binary-2.8.7.dev0-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
ERROR: psycopg2_binary-2.8.7.dev0-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl is not a supported wheel on this platform.

python3 -c"from packaging import tags; print('\n'.join([str(t) for t in tags.sys_tags()]))" |head -5
Requirement already satisfied: packaging in ./venvs/testing/lib/python3.8/site-packages (20.8)
Requirement already satisfied: pyparsing>=2.0.2 in ./venvs/testing/lib/python3.8/site-packages (from packaging) (2.4.7)
cp38-cp38-macosx_11_0_arm64
cp38-cp38-macosx_11_0_universal2
cp38-abi3-macosx_11_0_arm64
cp38-abi3-macosx_11_0_universal2
cp38-none-macosx_11_0_arm64

[bschollnick]

Is there anything else that I can do to help?

[bschollnick]

FYI, I meant to post this here. But I ended up reposting it back to the same Pillow Thread (too many windows open at once).

Okay, obviously I'm missing something here.

_

(testing) dist % pip install psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl
ERROR: psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl is not a supported wheel on this platform.
From a comment from Pillow:

I build Pillow and jpegsrc-6b from source before filing this issue and that didn't need any changes other than specifying "-arch arm64 -arch x86_64" when building the latter. That said, I haven't tested the intel part of the wheel yet (but I don't expect problems there, given past experience with the ppc to intel transition).

_

So I probably need to change the target build architecture, but I don't see where to do that... ?

[dvarrazzo]

Hello @bschollnick

I think you should hear from the people who contributed to pypa/pip#9138

I have no details about the M1 architecture and the specific details to address to build packages there.

[awildjosh]

As a quick aside, thank you both very much for your efforts :^]

If anyone with more know-how would like to remote in on my machine to further troubleshoot, let me know