classification
Title: macOS Python builds from Python.org ignore DYLD_LIBRARY_PATH due to hardened runtime
Type: Stage: resolved
Components: macOS Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: dgelessus, ned.deily, ronaldoussoren
Priority: normal Keywords:

Created on 2020-04-05 20:20 by dgelessus, last changed 2020-04-20 20:25 by dgelessus. This issue is now closed.

Messages (3)
msg365824 - (view) Author: (dgelessus) * Date: 2020-04-05 20:20
Recent Python.org versions of Python for macOS no longer respect the DYLD_LIBRARY_PATH environment variable for extending the dynamic library search path, and the envvar is completely invisible to the Python process. This is the case since at least Python 3.7.7 and Python 3.8.2. It was *not* the case with Python 3.7.5 or Python 3.8.0 or any earlier versions (I haven't tested 3.7.6 and 3.8.1). For example:

$ python3.6 --version
Python 3.6.8
$ DYLD_LIBRARY_PATH=tests/objc python3.6 -c 'import os; print(os.environ.get("DYLD_LIBRARY_PATH"))'
tests/objc
$ python3.7 --version
Python 3.7.7
$ DYLD_LIBRARY_PATH=tests/objc python3.7 -c 'import os; print(os.environ.get("DYLD_LIBRARY_PATH"))'
None

This seems to be because the Python binaries now fulfill the requirements for notarization (as mentioned in https://www.python.org/downloads/release/python-377/#macos-users), which includes enabling the hardened runtime (https://developer.apple.com/documentation/security/hardened_runtime), which by default hides DYLD_LIBRARY_PATH (and other DYLD_... envvars) from the hardened binary.

To disable this protection and allow using DYLD_... envvars again, the entitlement com.apple.security.cs.allow-dyld-environment-variables (https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-dyld-environment-variables) can be added to a hardened binary. The Python binaries seem to have some entitlements, but not .allow-dyld-environment-variables:

$ codesign --display --entitlements=:- python3.7
Executable=/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.cs.disable-executable-page-protection</key>
<true/>
</dict>
</plist>

Would it be possible to add this entitlement to the Python binaries, so that DYLD_LIBRARY_PATH can be used again, as was possible in previous versions?
msg366808 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-20 02:02
Thanks for the report. Yes, the choice of entitlements to request whith code hardening was somewhat arbitrary; we tried to keep them to a minimum.  I didn't anticipate that the DYLD_ env variables would be used much, sorry about that. Future python.org macOS installers will include that entitlement, starting with 2.7.18 (the final 2.7 release), 3.7.8, 3.8.3, and 3.9.0a6.
msg366877 - (view) Author: (dgelessus) * Date: 2020-04-20 20:25
I can confirm that the newly released Python 2.7.18 has the .allow-dyld-environment-variables entitlement:

$ ./python2.7 --version
Python 2.7.18
$ codesign --display --entitlements=:- python2.7
Executable=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.cs.disable-executable-page-protection</key>
<true/>
</dict>
</plist>

and accepts DYLD_LIBRARY_PATH again:

$ DYLD_LIBRARY_PATH=tests/objc ./python2.7 -c 'import os; print(os.environ.get("DYLD_LIBRARY_PATH"))'
tests/objc

Thank you for the fix!
History
Date User Action Args
2020-04-20 20:25:04dgelessussetmessages: + msg366877
2020-04-20 02:02:27ned.deilysetstatus: open -> closed
versions: + Python 2.7
messages: + msg366808

resolution: fixed
stage: resolved
2020-04-06 03:18:22ned.deilysetassignee: ned.deily
versions: + Python 3.9
2020-04-05 20:20:49dgelessuscreate