classification
Title: ImportError: Symbol not found: __PyCodecInfo_GetIncrementalDecoder when building 2.7.x on macOS
Type: compile error Stage: test needed
Components: Build Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ajstewart, ned.deily
Priority: low Keywords:

Created on 2017-03-18 14:35 by ajstewart, last changed 2017-03-20 19:06 by ned.deily.

Files
File name Uploaded Description Edit
spack-build.out ajstewart, 2017-03-18 14:35 Python build log
python-config-vars.txt ajstewart, 2017-03-19 20:21 Output of sysconfig.get_config_vars()
dyld_print_libraries.txt ajstewart, 2017-03-19 20:33 DYLD_PRINT_LIBRARIES
dyld_print_libraries.txt ajstewart, 2017-03-19 20:43 New DYLD_PRINT_LIBRARIES output
Messages (9)
msg289815 - (view) Author: Adam Stewart (ajstewart) Date: 2017-03-18 14:35
I'm trying to build Python 2.7.13 with clang on macOS 10.12.3 with the Spack package manager, but it fails to build the _io module. The exact error message from the build log can be seen here:

https://github.com/LLNL/spack/issues/3478#issuecomment-287548431

This only seems to occur for me on macOS; I can't reproduce it on Linux. I checked my environment, but I don't have any Python-related environment variables, nor do I have any variables like DYLD_LIBRARY_PATH set that could cause problems. I'm a developer for the Spack package manager, so any problems that you help me solve will greatly benefit our community.

I have attached the build log. Please let me know if there is any more information I can provide you with.
msg289822 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-18 17:25
I'm not familiar with Spack and I'm certainly not going to try to deeply analyze a build that inserts *that* many separate -I and -L options into the compiler calls :)  But there have been a number of reports of problems with this symptom in the past (try searching the web) and at least one previous issue opened here (Issue26498).  The root cause in all of these is that at some point there is a mismatch between a python executable of one 2.7.x version being dynamically linked at run time with python shared libraries of a different 2.7.x version.  While a python build should guard against this happening, often it doesn't matter when such a mismatch occurs.  However, there was an incompatibility introduced in the lifetime of 2.7.x involving __PyCodecInfo_GetIncrementalDecoder (I forget the details).  My guess is that you will find that the problem goes away if you avoid the use of '--enable-shared' on the Python configure; most Python distributions on macOS use --enable-framework rather than --enable-shared and we seldom test it.  Or use neither.  You should also check that there are no third-party Python 2.7's on PATH and/or Python 2.7 shared libraries on the dyld library search paths, other than the Apple-supplied pythons in /usr/bin.  It would be nice to finally figure out and resolve where the hole is but I've never been able to reproduce the failure myself.
msg289827 - (view) Author: Adam Stewart (ajstewart) Date: 2017-03-18 20:12
> I'm certainly not going to try to deeply analyze a build that inserts *that* many separate -I and -L options into the compiler calls

I believe those were necessary to get the build working. Spack doesn't install anything into /usr/, and without those flags, Python doesn't know where to search for its dependencies. I tried removing them anyway but it didn't help.

> My guess is that you will find that the problem goes away if you avoid the use of '--enable-shared' on the Python configure

Just tried this and it does indeed go away! I can try convincing the other developers that this is the best option.

> most Python distributions on macOS use --enable-framework rather than --enable-shared and we seldom test it.

I just tried using using `--enable-framework` and `make install` failed. Seeing lots of `ImportError: No module named site` errors.

> You should also check that there are no third-party Python 2.7's on PATH and/or Python 2.7 shared libraries on the dyld library search paths, other than the Apple-supplied pythons in /usr/bin.

This could very well be the cause of the problem. I happen to have both Python 2 and 3 installed with both Homebrew and Anaconda, and all 4 are in my PATH. I don't have any PYTHON or LD/DYLD related env vars though. If you think it's worth testing, I can try removing everything from my PATH except /usr/bin:/bin
msg289860 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-19 19:19
Glad the --enable-shared workaround worked around.  If you feel like exploring the issue further, a couple of things that might be useful would be to set one or more of dyld's debugging environment variables, like DYLD_PRINT_LIBRARIES, to try to see exactly which shared library is being accessed by mistake and at what point, assuming that is the root cause; see man (1) dyld for more info.  My guess is that you'll see a Homebrew or Anaconda Python library from /usr/local/lib or some such being dynamically loaded in error.  Another option might be to temporarily mv rename any such Python libraries under /usr/local/lib and see if that makes a difference.  As I said, it would be nice to finally put this issue to bed but, it's tough to do that without being to able to reliably reproduce it and, at this point, it doesn't seem to affect enough users to make it very high priority.
msg289861 - (view) Author: Adam Stewart (ajstewart) Date: 2017-03-19 20:21
I agree, the `--enable-shared` fix is nice, but I would also love to squash this bug.

An interesting piece of information: When I provide the full path to the executable (not just relying on the hashed python location), I see:

$ /Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/bin/python
Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.version
2.7.10 (default, Jul 30 2016, 19:40:32) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
>>> print sys.path
['', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python27.zip', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7/plat-darwin', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7/plat-mac', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7/lib-tk', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7/lib-old', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7/lib-dynload', '/Users/Adam/.local/lib/python2.7/site-packages', '/Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/lib/python2.7/site-packages']

Note that I am building Python 2.7.13, not 2.7.10. Homebrew installs Python in /usr/local and that version is 2.7.12. My Ananconda installation is in ~/anaconda2 and is 2.7.13. However, the system Python installed at /usr is 2.7.10. Perhaps this is where it is picking up libraries from?

Here's some more debug information that could possibly be helpful. I have attached the output of:

$ /Users/Adam/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/python-2.7.13-tjqn5npfo573q2jw3dumzhzctfd2vvdv/bin/python -c 'from __future__ import print_function; import sysconfig; vars = sysconfig.get_config_vars(); [print(var, vars[var]) for var in vars]' > ~/python-config-vars.txt

Nothing in there looks particularly suspicious to me, but maybe you can find something fishy. I'll try out DYLD_PRINT_LIBRARIES next.
msg289862 - (view) Author: Adam Stewart (ajstewart) Date: 2017-03-19 20:33
The output of DYLD_PRINT_LIBRARIES is attached as well.
msg289863 - (view) Author: Adam Stewart (ajstewart) Date: 2017-03-19 20:43
Made a mistake. Please see the new output of DYLD_PRINT_LIBRARIES. This time I actually ran a Python command to see which libraries were loaded.
msg289864 - (view) Author: Adam Stewart (ajstewart) Date: 2017-03-19 21:44
I also tried building with a Homebrew-installed GCC 6.2.0 but that had the same result.
msg289900 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-20 19:05
Thanks, Adam!  Note to self: see Makefile changes in PR 737 for Issue15590.
History
Date User Action Args
2017-03-20 19:06:00ned.deilysetmessages: - msg289899
2017-03-20 19:05:46ned.deilysetmessages: + msg289900
2017-03-20 19:04:27ned.deilysetmessages: + msg289899
2017-03-19 21:44:35ajstewartsetmessages: + msg289864
2017-03-19 20:43:59ajstewartsetfiles: + dyld_print_libraries.txt

messages: + msg289863
2017-03-19 20:33:11ajstewartsetfiles: + dyld_print_libraries.txt

messages: + msg289862
2017-03-19 20:21:24ajstewartsetfiles: + python-config-vars.txt

messages: + msg289861
2017-03-19 19:19:15ned.deilysetpriority: normal -> low

messages: + msg289860
stage: test needed
2017-03-18 20:12:08ajstewartsetmessages: + msg289827
2017-03-18 17:29:21ned.deilysettitle: ImportError: No module named _io -> ImportError: Symbol not found: __PyCodecInfo_GetIncrementalDecoder when building 2.7.x on macOS
2017-03-18 17:25:52ned.deilysetnosy: + ned.deily
messages: + msg289822
2017-03-18 14:35:07ajstewartcreate