This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: dylibs not loading properly from NFS mounts
Type: behavior Stage: resolved
Components: macOS Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: embassy_vfx, ned.deily, ronaldoussoren
Priority: normal Keywords:

Created on 2022-01-26 21:05 by embassy_vfx, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (4)
msg411793 - (view) Author: (embassy_vfx) Date: 2022-01-26 21:05
MacOS Catalina 10.15.4 and 10.15.7
tested with XCode Python3.8 install and Python3.7.9 installer from https://www.python.org/ftp/python/3.7.9/python-3.7.9-macosx10.9.pkg (and my own build of 3.7.12)

When importing libraries that use shared object files to an NFS mounted directory, they do not open the dynamic library properly. We can use PySide2 as an example here. This workflow works fine in python 2.7 using the same steps.

Repro Steps:
1)pip install PySide2 to an NFS mounted directory: `/usr/local/bin/python3.7 -m pip install --target=/<path_to_nfs_dir>/nfs_3.7 PySide2`
2)`export PYTHONPATH=/<path_to_nfs_dir>/nfs_3.7`
3)run python `/usr/local/bin/python3.7`
4)import PySide2 - you should get an ImportError that looks like "dlopen(/<path_to_nfs_dir>/nfs_3.7/shiboken2/shiboken2.abi3.so, 2): Library not loaded: @rpath/libshiboken2.abi3.5.15.dylib" - otool shows shiboken2.abi3.so has it's rpath set as @loader_path and I've confirmed both files exist.

If I follow the same procedure above but instead install PySide2 to `/tmp/local_3.7` everything works as expected and the binaries are loaded properly.

I had this same issue with standard library .so files when I installed my custom build of python to an NFS mounted directory (_ssl as an example) but only when running through a virtual environment. Possibly @rath/@loader_path related? This seems to only be an issue on MacOS as I have no issues on the same NFS mount on Linux.
msg412189 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2022-01-31 01:48
Thanks for the report but there are a lot of variables here that make this very difficult to act on. First, there have been many changes in macOS 10.15 regarding file permissions and file system layouts (with APFS et al) and further changes in the more recent versions of macOS, 11 Big Sur and now 12 Monterey. Further, Python 3.7 is now in the security-fix-only phase of its life cycle which means we no longer produce binary installers for macOS or Windows for 3.7. 3.7.9 was released prior to the official release of macOS 11 and so, as noted on the changelog for 3.7.9, macOS 11+ is not fully supported by 3.7.x. We have also learned more about these operating systems since then and there are changes in Python and in how we manufacture python.org macOS installers that are reflected in current releases of Python 3.9.x and 3.10.x. Beyond that, AFAIK, we do not do any testing of macOS NFS clients or servers and have no setup for that at hand. And throwing in a third-party extension module (shiboken2.abi3.so) plus use of @rpath on top of all that adds even more variability.

My suggestion is to try to reproduce this with a current, fully-supported python.org macOS installer download (i.e. the 3.10.2 universal2 installer) and preferably on a current version of macOS, currently 12.2 or possibly 11.6.3, and also check "System Preferences" -> "Security & Privacy" -> "Privacy settings" like "Full Disk Access" and "Files and Folders"; if problems persist, try doing some dynamic loader debugging using the DYLD_* environment variables documented in the macOS dyld man page (man dyld). Otherwise, as your issue stands now, it seems unlikely that anyone here would be willing to invest the significant time to try to reproduce and debug what seems to be an unusual macOS configuration with out-of-date OS and Python versions.

Perhaps a simpler option is to just avoid use of NFS file systems on macOS for this purpose.

Sorry I don't have a more positive answer for you.
msg412306 - (view) Author: (embassy_vfx) Date: 2022-02-01 20:55
Hi Ned,

Thanks for the reply. I figured that this may be the case but wanted to
report it anyway.

I currently do not have access to a more up-to-date Mac OS but once I do I
will test and report anything unusual back. I also have not tested other
file systems but will most likely do that as well. Thanks again for your
time here.

Kenny

On Sun, Jan 30, 2022 at 5:48 PM Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily <nad@python.org> added the comment:
>
> Thanks for the report but there are a lot of variables here that make this
> very difficult to act on. First, there have been many changes in macOS
> 10.15 regarding file permissions and file system layouts (with APFS et al)
> and further changes in the more recent versions of macOS, 11 Big Sur and
> now 12 Monterey. Further, Python 3.7 is now in the security-fix-only phase
> of its life cycle which means we no longer produce binary installers for
> macOS or Windows for 3.7. 3.7.9 was released prior to the official release
> of macOS 11 and so, as noted on the changelog for 3.7.9, macOS 11+ is not
> fully supported by 3.7.x. We have also learned more about these operating
> systems since then and there are changes in Python and in how we
> manufacture python.org macOS installers that are reflected in current
> releases of Python 3.9.x and 3.10.x. Beyond that, AFAIK, we do not do any
> testing of macOS NFS clients or servers and have no setup for that at hand.
> And throwing in a third-party extension module (shiboken2.abi3.so) plus
> use of @rpath on top of all that adds even more variability.
>
> My suggestion is to try to reproduce this with a current, fully-supported
> python.org macOS installer download (i.e. the 3.10.2 universal2
> installer) and preferably on a current version of macOS, currently 12.2 or
> possibly 11.6.3, and also check "System Preferences" -> "Security &
> Privacy" -> "Privacy settings" like "Full Disk Access" and "Files and
> Folders"; if problems persist, try doing some dynamic loader debugging
> using the DYLD_* environment variables documented in the macOS dyld man
> page (man dyld). Otherwise, as your issue stands now, it seems unlikely
> that anyone here would be willing to invest the significant time to try to
> reproduce and debug what seems to be an unusual macOS configuration with
> out-of-date OS and Python versions.
>
> Perhaps a simpler option is to just avoid use of NFS file systems on macOS
> for this purpose.
>
> Sorry I don't have a more positive answer for you.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue46540>
> _______________________________________
>
msg412307 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2022-02-01 21:17
> I currently do not have access to a more up-to-date Mac OS but once I do I
will test and report anything unusual back.

Good luck! Since there's a good chance that this issue will languish in our open backlog, I am going to mark it as closed for now. But please re-open this issue if you find anything that appears to be an issue with Python rather than just macOS. Two other suggestions for debugging:
1. You've probably already done this but double-check file and directory access permissions on the NFS-mounted file system up to root. We have seen other issues with relative paths on 10.15+ with the introduction of the read-only system volume and related file system layout complications.
2. You might try using install_name_tool to alter copies of those third-party .so(s) and dylib(s) that use @rpath and instead use an absolute path to their installed location(s). That would eliminate one variable.
History
Date User Action Args
2022-04-11 14:59:55adminsetgithub: 90698
2022-02-01 21:17:10ned.deilysetstatus: open -> closed
resolution: third party
messages: + msg412307

stage: resolved
2022-02-01 20:55:18embassy_vfxsetmessages: + msg412306
2022-01-31 01:48:45ned.deilysetmessages: + msg412189
2022-01-26 21:05:40embassy_vfxcreate