Python 3.7.9 and openfst

[meliksahturker]

I installed via pip in Python 3.7.9 and wanted to test Analyzer.

Trying to import analyze with:
from turkish_morphology import analyze
results in
ImportError: cannot import name 'pywrapfst' from 'external.org_openfst' (unknown location)
for Windows,

and
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/external/org_openfst/pywrapfst.so, 2): no suitable image found. Did find: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/external/org_openfst/pywrapfst.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03 /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/external/org_openfst/pywrapfst.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03

for Mac even though I installed openfst via brew.

Installing openfst via pip fails in both OS.

So my question is: is Python 3.9 is required and neccesary to run turkish-morphology especially considering these errors related to openfst?

[alerque]

I believe this came up in another recent issue report and yes Python 3.9 is a minimum requirement. This should probably be better documented and/or noted in the project setup so it gives a proper error up front rather than dying later.

[meliksahturker]

I just tested with Python 3.9.0 and still end up with the same error. I am looking for proper ways to install openfst that would relax this error. I would be very happy to hear any advice in this regard.

[meliksahturker]

Can you specify what version and what configuration of openfst do I need to install? I really want to test turkish-morphology but I am having huge headache with openfst

[ozturel]

In fact, you don't need to install openfst alongside the turkish-morphology package at all. The Python wrapper for OpenFst is shipped together with the pypi release, see:

build rule: https://github.com/google-research/turkish-morphology/blob/master/third_party/openfst.BUILD#L1188-L1207
data
package data: https://github.com/google-research/turkish-morphology/blob/master/setup.py#L39-L44

Confirming @alerque, if you want to install turkish-morphology through pip the only requirement is Python 3.9 (since some of the external dependencies of turkish-morphology requires >=Python3.9 now on).

I tested both the source and wheel distributions from pypi on linux and I cannot reproduce the failure. I currently don't have immediate access to a Windows or Mac OS machine, therefore it would take some time for me to look into it.

One other alternative that you can try right away is to build turkish-morphology from source. For that minimum requirements are Bazel 4.1.0 and Python3.9, see:

https://github.com/google-research/turkish-morphology#requirements

To install from source, once you clone the git repo, run below from the project root directory (preferably within a Python virtual environment):

bazel build //...
bazel-bin/setup install

[meliksahturker]

I installed through pip on Python 3.9.6 and MacOS 11.2.3 in a fresh virtualenv.
The error I get is the same:
ImportError: dlopen(/Users/meliksahturker/Desktop/py39/lib/python3.9/site-packages/external/org_openfst/pywrapfst.so, 2): no suitable image found. Did find: /Users/meliksahturker/Desktop/py39/lib/python3.9/site-packages/external/org_openfst/pywrapfst.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03 /Users/meliksahturker/Desktop/py39/lib/python3.9/site-packages/external/org_openfst/pywrapfst.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03

I'll try bazel one.

[ozturel]

I see. It might be the case that pywrapfst.so is platform dependent and if that is the case it should not be included in the sdist source distribution. I'll check whether we can include the openfst Cython extension while building the source package.

[meliksahturker]

Bazel 4.1.0 failed building, too.
Here is the result in case you're interested in it:

ERROR: /private/var/tmp/_bazel_meliksahturker/d47feaeed19322435d08c8806c4db040/external/org_opengrm_thrax/BUILD.bazel:184:11: @org_opengrm_thrax//:regression_test-lib depends on @com_google_googletest//:gtest in repository @com_google_googletest which failed to fetch. no such package '@com_google_googletest//': java.io.IOException: Error downloading [https://github.com/google/googletest/archive/master.zip] to /private/var/tmp/_bazel_meliksahturker/d47feaeed19322435d08c8806c4db040/external/com_google_googletest/temp6292830808643682562/master.zip: Checksum was 5eb0fbf38d4973f90e5c4f264b17dc8dc45a3be87ad737ced855128ccb79e67f but wanted aa9efa9edaa3b06868c836666bb7b8901b084c9210746111565a02973cbb6d20 ERROR: Analysis of target '//src/analyzer/morphophonemics:consonant_change_test' failed; build aborted: Analysis failed INFO: Elapsed time: 8.010s INFO: 0 processes. FAILED: Build did NOT complete successfully (39 packages loaded, 976 targets c\ onfigured)

[ozturel]

I have fixed the checksums. Could you try again installing from source using Bazel after cloning the current master branch?

[meliksahturker]

New error:

`INFO: Analyzed 76 targets (1 packages loaded, 3 targets configured).
INFO: Found 76 targets...
ERROR: /private/var/tmp/_bazel_meliksahturker/d47feaeed19322435d08c8806c4db040/external/org_opengrm_thrax/BUILD.bazel:160:10: Compiling src/bin/compiler.cc [for host] failed: (Exit 1): cc_wrapper.sh failed: error executing command external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 39 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from external/org_opengrm_thrax/src/bin/compiler.cc:17:
In file included from external/org_opengrm_thrax/src/include/thrax/compiler.h:20:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-compiler.h:32:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-manager.h:28:
In file included from external/org_opengrm_thrax/src/include/thrax/abstract-grm-manager.h:29:
In file included from /usr/local/include/fst/extensions/mpdt/compose.h:25:
/usr/local/include/fst/extensions/mpdt/mpdt.h:139:29: error: no template named 'make_unique' in namespace 'fst'; did you mean 'std::make_unique'?
return fst::make_unique<PdtStack<StackId, Label>>(*ptr);
^~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:2753:1: note: 'std::make_unique' declared here
make_unique(_Args&&... __args)
^
In file included from external/org_opengrm_thrax/src/bin/compiler.cc:17:
In file included from external/org_opengrm_thrax/src/include/thrax/compiler.h:20:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-compiler.h:32:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-manager.h:28:
In file included from external/org_opengrm_thrax/src/include/thrax/abstract-grm-manager.h:29:
In file included from /usr/local/include/fst/extensions/mpdt/compose.h:25:
/usr/local/include/fst/extensions/mpdt/mpdt.h:139:29: error: no template named 'make_unique' in namespace 'fst'; did you mean 'std::make_unique'?
return fst::make_unique<PdtStack<StackId, Label>>(*ptr);
^~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:2753:1: note: 'std::make_unique' declared here
make_unique(_Args&&... __args)
^
In file included from external/org_opengrm_thrax/src/bin/compiler.cc:17:
In file included from external/org_opengrm_thrax/src/include/thrax/compiler.h:20:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-compiler.h:32:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-manager.h:28:
In file included from external/org_opengrm_thrax/src/include/thrax/abstract-grm-manager.h:29:
In file included from /usr/local/include/fst/extensions/mpdt/compose.h:25:
/usr/local/include/fst/extensions/mpdt/mpdt.h:300:9: error: no template named 'make_unique' in namespace 'fst'; did you mean 'std::make_unique'?
fst::make_unique<PdtStack<StackId, Label>>(vectors[level]);
^~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:2753:1: note: 'std::make_unique' declared here
make_unique(_Args&&... __args)
^
In file included from external/org_opengrm_thrax/src/bin/compiler.cc:17:
In file included from external/org_opengrm_thrax/src/include/thrax/compiler.h:20:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-compiler.h:32:
In file included from external/org_opengrm_thrax/src/include/thrax/grm-manager.h:28:
In file included from external/org_opengrm_thrax/src/include/thrax/abstract-grm-manager.h:29:
In file included from /usr/local/include/fst/extensions/mpdt/compose.h:25:
/usr/local/include/fst/extensions/mpdt/mpdt.h:300:9: error: no template named 'make_unique' in namespace 'fst'; did you mean 'std::make_unique'?
fst::make_unique<PdtStack<StackId, Label>>(vectors[level]);
^~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:2753:1: note: 'std::make_unique' declared here
make_unique(_Args&&... __args)
^
4 errors generated.
INFO: Elapsed time: 84.701s, Critical Path: 18.50s
INFO: 408 processes: 26 internal, 382 darwin-sandbox.
FAILED: Build did NOT complete successfully`

[mezig351]

I installed via pip in Python 3.7.9 and wanted to test Analyzer.

Trying to import analyze with:
from turkish_morphology import analyze
results in
ImportError: cannot import name 'pywrapfst' from 'external.org_openfst' (unknown location)
for Windows,

and
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/external/org_openfst/pywrapfst.so, 2): no suitable image found. Did find: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/external/org_openfst/pywrapfst.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03 /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/external/org_openfst/pywrapfst.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03

for Mac even though I installed openfst via brew.

Installing openfst via pip fails in both OS.

So my question is: is Python 3.9 is required and neccesary to run turkish-morphology especially considering these errors related to openfst?

I get the same error, additionally I tried installing openfst with conda. But I see that was not necessary.

[meliksahturker]

Still having interest, bringing this up.
Is there any plan to fix?

[iesener]

I'm having the same problem as above. I use Python 3.10 and Bazel 5.0 on MacOS. Does it work on Linux? Should I try it on a Linux? I'd appreciate any help. Thanks.

from turkish_morphology import analyze

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/turkish_morphology/analyze.py", line 20, in <module>
    from turkish_morphology import fst
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/turkish_morphology/fst.py", line 22, in <module>
    from external.org_openfst import pywrapfst
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/external/org_openfst/pywrapfst.so, 0x0002): tried: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/external/org_openfst/pywrapfst.so' (not a mach-o file), '/usr/lib/pywrapfst.so' (no such file)