Author andrewfg1992
Recipients andrewfg1992, ned.deily
Date 2020-08-05.19:11:04
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1596654664.79.0.569998092865.issue41129@roundup.psfhosted.org>
In-reply-to
Content
So I believe I've found the problem as to why we can't get the extension modules to build in our network area.

It seems to be a problem with setup.py's find_file() and is_macosx_sdk_path() functions when the extension modules are building. You can find these functions around line 182 of setup.py.

For some background, our network location /mathworks is usually mounted at root, but due to the restrictions on macOS Catalina, it mounts at /System/Volumes/Data and we then add a symlink to the root directory so "/mathworks -> /System/Volumes/Data/mathworks" as to not break old code and scripts. So the real problem is that we cannot find extension module files under /System/Volumes/Data.

This problem occurs because in setup.py, when we attempt to find an extension module file with find_file() wherein the file registers as being in the mac sdk path since the path starts with "/System". This is a problem because find_file() then prepends the file with the sdk path which isn't quite right and fails the existence check. For example a source file that actually exists at "/System/Volumes/Data/mathworks/hub/scratch/aflewell/Python-3.8.2/Modules/" find_file() calls os.path.exists() on "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Volumes/Data/mathworks/hub/scratch/aflewell/Python-3.8.2/Modules/" which fails since the path does not exist at or after the "Volumes" part. This causes find_file to think the file does not exist at all, so the compiler ends up trying to compile the file without an absolute path or relative path to it, which fails.

This can be fixed by changing is_macosx_sdk_path() so it does not consider /System/Volumes/Data as part of the sdk path:

Change from:

def is_macosx_sdk_path(path):
    """
    Returns True if 'path' can be located in an OSX SDK
    """
    return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
                or (path.startswith('/System/') )
                or path.startswith('/Library/') )

to

def is_macosx_sdk_path(path):
    """
    Returns True if 'path' can be located in an OSX SDK
    """
    return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
                or (path.startswith('/System/') and not path.startswith('/System/Volumes/Data'))
                or path.startswith('/Library/') )

This change prevents the sdk path from being prepended to the expected path to the file.

I'm not sure if this is the right fix, so I'd like your opinion on it. I don'r know much about mac's developer sdk and how it works, but it doesn't seem like anything under /System/Volumes should be included in or locatable in the sdk. Anyways, it would be nice to have an official fix for this in the official python source.

Regarding reproduction, you may be able to reproduce by trying to build python under /Systems/Volumes/Data, if not, it may have to do with the mounting of our network filesystem. Thanks for your help.
History
Date User Action Args
2020-08-05 19:11:04andrewfg1992setrecipients: + andrewfg1992, ned.deily
2020-08-05 19:11:04andrewfg1992setmessageid: <1596654664.79.0.569998092865.issue41129@roundup.psfhosted.org>
2020-08-05 19:11:04andrewfg1992linkissue41129 messages
2020-08-05 19:11:04andrewfg1992create