pip install pyfst fails on OSX Yosemite

[delip]

Strange that it can't seem to find the unordered_map header. Any suggestions?

gcc -fno-strict-aliasing -I//anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I//anaconda/include/python2.7 -c fst/_fst.cpp -o build/temp.macosx-10.5-x86_64-2.7/fst/_fst.o -O2
  In file included from fst/_fst.cpp:321:
  /Users/delip/prefix/include/fst/symbol-table.h:26:10: fatal error: 'unordered_map' file not found
  #include <unordered_map>
           ^
  1 error generated.
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for pyfst
Failed to build pyfst
Installing collected packages: pyfst
  Running setup.py install for pyfst
    Complete output from command //anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/qj/c2h005w56nlc2hpc5n1f8s5r0000gn/T/pip-build-UETMSM/pyfst/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/qj/c2h005w56nlc2hpc5n1f8s5r0000gn/T/pip-cq1YQe-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    running build_ext
    building 'fst._fst' extension
    gcc -fno-strict-aliasing -I//anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I//anaconda/include/python2.7 -c fst/_fst.cpp -o build/temp.macosx-10.5-x86_64-2.7/fst/_fst.o -O2
    In file included from fst/_fst.cpp:321:
    /Users/delip/prefix/include/fst/symbol-table.h:26:10: fatal error: 'unordered_map' file not found
    #include <unordered_map>
             ^
    1 error generated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "//anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/qj/c2h005w56nlc2hpc5n1f8s5r0000gn/T/pip-build-UETMSM/pyfst/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/qj/c2h005w56nlc2hpc5n1f8s5r0000gn/T/pip-cq1YQe-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/qj/c2h005w56nlc2hpc5n1f8s5r0000gn/T/pip-build-UETMSM/pyfst

[j9ac9k]

I just ran into this issue too, from what I can tell this is a compiler error, and you need to specify the use of a newer compiler (which you can install with homebrew).

[j9ac9k]

I made some progress on this issue, but it's still not fixed.

with homebrew install gcc4.7

brew tap homebrew/versions
brew install gcc47

Inside the setup.py file after the imports of modules add the following:

os.environ["CC"] = "gcc-4.7"
os.environ["CXX"] = "g++-4.7"

It now compiles/builds, however it still isn't functioning...

[delip]

can you expand on "it still isn't functioning.." ?

On Mon, Oct 12, 2015 at 2:42 AM, Ogi Moore [email protected] wrote:

I made some progress on this issue, but it's still not fixed.

with homebrew install gcc4.7

brew tap homebrew/versions
brew install gcc47

Inside the setup.py file after the imports of modules add the following:

os.environ["CC"] = "gcc-4.7"
os.environ["CXX"] = "g++-4.7"

It now compiles/builds, however it still isn't functioning...

—
Reply to this email directly or view it on GitHub
#24 (comment).

[j9ac9k]

After the compile finishes, I start python, and upon running import fst, I get errors about symbols missing from _fst.so Sorry I can't give a more accurate error, I'm at work right now, I'll follow up when I get home.

[j9ac9k]

Here is an example error I get after a successful compile of pyfst:

ogi@Mac-VM ~ ❯❯❯ ipython
Python 3.4.3 |Continuum Analytics, Inc.| (default, Mar  6 2015, 12:07:41)
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import fst
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-7279038b919c> in <module>()
----> 1 import fst

/Users/ogi/miniconda3/lib/python3.4/site-packages/pyfst-0.2.3.dev0-py3.4-macosx-                                                                    10.5-x86_64.egg/fst/__init__.py in <module>()
----> 1 from fst._fst import EPSILON, EPSILON_ID, SymbolTable,\
      2         read, read_log, read_std, read_symbols, \
      3         LogWeight, LogArc, LogState, LogVectorFst,\
      4         TropicalWeight, StdArc, StdState, StdVectorFst, read_symbols_tex                                                                    t
      5

ImportError: dlopen(/Users/ogi/miniconda3/lib/python3.4/site-packages/pyfst-0.2.                                                                    3.dev0-py3.4-macosx-10.5-x86_64.egg/fst/_fst.so, 2): Symbol not found: __ZTINSt8                                                                    ios_base7failureB5cxx11E
  Referenced from: /Users/ogi/miniconda3/lib/python3.4/site-packages/pyfst-0.2.3                                                                    .dev0-py3.4-macosx-10.5-x86_64.egg/fst/_fst.so
  Expected in: dynamic lookup

I can successfully build with later versions of gcc, but I get errors similar to this one.

Suggestions?

EDIT:

I should point out I'm using the UFAL-DSG fork of pyfst as it has been more recently updated; however development has stopped there as well.

[delip]

I'm guessing this is because of a Python version mismatch. I see you have
Python 3.4. Can you try with Python 2.7.x?

On Mon, Oct 12, 2015 at 7:30 PM, Ogi Moore [email protected] wrote:

Here is an example error I get after a successful compile of pyfst:

ogi@Mac-VM ~ ❯❯❯ ipython
Python 3.4.3 |Continuum Analytics, Inc.| (default, Mar 6 2015, 12:07:41)
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.

In [1]: import fst

ImportError Traceback (most recent call last)
in ()
----> 1 import fst

/Users/ogi/miniconda3/lib/python3.4/site-packages/pyfst-0.2.3.dev0-py3.4-macosx- 10.5-x86_64.egg/fst/init.py in ()
----> 1 from fst._fst import EPSILON, EPSILON_ID, SymbolTable,
2 read, read_log, read_std, read_symbols,
3 LogWeight, LogArc, LogState, LogVectorFst,
4 TropicalWeight, StdArc, StdState, StdVectorFst, read_symbols_tex t
5

ImportError: dlopen(/Users/ogi/miniconda3/lib/python3.4/site-packages/pyfst-0.2. 3.dev0-py3.4-macosx-10.5-x86_64.egg/fst/_fst.so, 2): Symbol not found: __ZTINSt8 ios_base7failureB5cxx11E
Referenced from: /Users/ogi/miniconda3/lib/python3.4/site-packages/pyfst-0.2.3 .dev0-py3.4-macosx-10.5-x86_64.egg/fst/_fst.so
Expected in: dynamic lookup

I can successfully build with later versions of gcc, but I get errors
similar to this one.

Suggestions?

—
Reply to this email directly or view it on GitHub
#24 (comment).

[j9ac9k]

I can try it later; however I do have pyfst working* on on my linux system with python 3. Again, note that I am using this fork: https://github.com/UFAL-DSG/pyfst of pyfst (not this one).

** for some reason the plotting within the jupyter notebook doesn't work with python 3, but works with python 2.

[j9ac9k]

I got it working!

there are a number of steps to be done here, not sure if they're all necessary but this is what I did

1. brew uninstall openfst
2. brew install gcc49
3. brew install openfst --build-from-source --cc=gcc-4.9 -cxx=g++-4.9
4. git clone https://github.com/UFAL-DSG/pyfst.git
5. nano ./pyfst/setup.py

⋅⋅1. near the top of the file, after the imports, add the following two lines of code:

os.environ["CC"] = "gcc-4.9"
os.environ["CXX"] = "g++-4.9"

1. set the environmental variables as described here: https://github.com/UFAL-DSG/pyfst
2. sudo python ./pyfst/setup.py install
3. sudo mv /usr/lib/libstdc++.6.dylib libstdc++.6.dylib.old
4. sudo cp /usr/local/Cellar/gcc49/4.9.3/lib/gcc/4.9/libstdc++.6.dylib /usr/lib/libstdc++.6.dylib

you will then be able to start a python kernel and import fst

[delip]

This is great. Well done! Pyfst is notoriously hard to install. It will be
great to turn this into a docker image.

On Wed, Oct 14, 2015 at 1:47 AM, Ogi Moore [email protected] wrote:

I got it working!

there are a number of steps to be done here, not sure if they're all
necessary but this is what I did

1. brew uninstall openfst 2, brew install gcc49 3, brew install
   openfst --build-from-source --cc=gcc-4.9 -cxx=g++-4.9
2. git clone https://github.com/UFAL-DSG/pyfst.git
3. nano ./pyfst/setup.py ⋅⋅*near the top of the file, after the
   imports, add the following two lines of code:

os.environ["CC"] = "gcc-4.9"
os.environ["CXX"] = "g++-4.9"

1. set the environmental variables as described here:
   https://github.com/UFAL-DSG/pyfst
2. sudo python ./pyfst/setup.py install
3. sudo mv /usr/lib/libstdc++.6.dylib libstdc++.6.dylib.old
4. sudo cp /usr/local/Cellar/gcc49/4.9.3/lib/gcc/4.9/libstdc++.6.dylib
   /usr/lib/libstdc++.6.dylib

you will then be able to start a python kernel and import fst

—
Reply to this email directly or view it on GitHub
#24 (comment).

[j9ac9k]

Not sure how a python library would work in a docker image on its own,but
I'm not a docker pro so I would be the last person to know how well that
would work.

There is still the IPython/Jupyter plotting issue that occurs when running
pyfst in Python 3. Looking through the code with the debugger, it's clear
that I'm in over my head.
On Oct 15, 2015 10:13 AM, "Delip Rao" [email protected] wrote:

This is great. Well done! Pyfst is notoriously hard to install. It will be
great to turn this into a docker image.

On Wed, Oct 14, 2015 at 1:47 AM, Ogi Moore [email protected]
wrote:

I got it working!

there are a number of steps to be done here, not sure if they're all
necessary but this is what I did

1. brew uninstall openfst 2, brew install gcc49 3, brew install
   openfst --build-from-source --cc=gcc-4.9 -cxx=g++-4.9
2. git clone https://github.com/UFAL-DSG/pyfst.git
3. nano ./pyfst/setup.py ⋅⋅*near the top of the file, after the
   imports, add the following two lines of code:

os.environ["CC"] = "gcc-4.9"
os.environ["CXX"] = "g++-4.9"

1. set the environmental variables as described here:
   https://github.com/UFAL-DSG/pyfst
2. sudo python ./pyfst/setup.py install
3. sudo mv /usr/lib/libstdc++.6.dylib libstdc++.6.dylib.old
4. sudo cp /usr/local/Cellar/gcc49/4.9.3/lib/gcc/4.9/libstdc++.6.dylib
   /usr/lib/libstdc++.6.dylib

you will then be able to start a python kernel and import fst

—
Reply to this email directly or view it on GitHub
#24 (comment).

—
Reply to this email directly or view it on GitHub
#24 (comment).

[delip]

My only worry is steps 3&4.

1. sudo mv /usr/lib/libstdc++.6.dylib libstdc++.6.dylib.old
2. sudo cp /usr/local/Cellar/gcc49/4.9.3/lib/gcc/4.9/libstdc++.6.dylib
   /usr/lib/libstdc++.6.dylib

This could cause other builds to fail. Did you try using LD_LIBRARY_PATH
instead of changing these files?

On Thu, Oct 15, 2015 at 1:12 PM, Delip Rao [email protected] wrote:

This is great. Well done! Pyfst is notoriously hard to install. It will be
great to turn this into a docker image.

On Wed, Oct 14, 2015 at 1:47 AM, Ogi Moore [email protected]
wrote:

I got it working!

there are a number of steps to be done here, not sure if they're all
necessary but this is what I did

1. brew uninstall openfst 2, brew install gcc49 3, brew install
   openfst --build-from-source --cc=gcc-4.9 -cxx=g++-4.9
2. git clone https://github.com/UFAL-DSG/pyfst.git
3. nano ./pyfst/setup.py ⋅⋅*near the top of the file, after the
   imports, add the following two lines of code:

os.environ["CC"] = "gcc-4.9"
os.environ["CXX"] = "g++-4.9"

1. set the environmental variables as described here:
   https://github.com/UFAL-DSG/pyfst
2. sudo python ./pyfst/setup.py install
3. sudo mv /usr/lib/libstdc++.6.dylib libstdc++.6.dylib.old
4. sudo cp
   /usr/local/Cellar/gcc49/4.9.3/lib/gcc/4.9/libstdc++.6.dylib
   /usr/lib/libstdc++.6.dylib

you will then be able to start a python kernel and import fst

—
Reply to this email directly or view it on GitHub
#24 (comment).

[j9ac9k]

So after talking with one of the developers for openfst, it turns out that openfst now (as of 1.5.0) supports python bindings, you just need to add --enable-python in the compile options.

Alternatively, you can perform a "pip install openfst" as well; however a lot of the same issues we've encountered here (won't compile, symbol not found, etc) are still occurring.

I'm shifting my efforts to getting the openfst python bindings working as pyfst development seems to have been abandoned.