classification
Title: [Windows] Can't import extension modules resolved via relative paths in sys.path
Type: behavior Stage: resolved
Components: Interpreter Core, Library (Lib), Windows Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: brett.cannon, eryksun, gregory.p.smith, neonene, orsenthil, paul.moore, smunday, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2021-02-02 17:03 by smunday, last changed 2021-06-14 06:28 by neonene. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25121 merged steve.dower, 2021-03-31 19:36
PR 25237 merged steve.dower, 2021-04-07 00:30
PR 25318 merged steve.dower, 2021-04-09 19:38
Messages (13)
msg386155 - (view) Author: Simon Munday (smunday) Date: 2021-02-02 17:03
If I attempt to import an extension module via something like this:

from pkg import extmodule

and it happens that "pkg" is found in a folder that is given in sys.path as a relative path, then the import fails, with 

ImportError: DLL load failed while importing extmodule: The parameter is incorrect.

This only happens with 3.8 and later.  

AFAICS the reason is that if the module resolution is done via a relative path, it results in dynload_win.c:_PyImport_FindSharedFuncptrWindows attempting to feed LoadLibraryExW a relative path for the .pyd file.   But LoadLibraryExW treats relative paths completely differently from absolute ones, in that it searches for the given path relative to the library search path entries rather than simply loading the DLL at that path.  But as of 3.8 the current directory is removed from the search path, so the .pyd is never found.  Since Python knows the specific file it wants LoadLibraryExW to load, having just resolved it via the import mechanism, I guess it ought to ensure it only ever calls LoadLibraryExW with an absolute path.  

(I'm assuming use of relative paths in sys.path is officially supported, since nothing I've found in the docs says otherwise.)
msg386898 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-13 03:14
> and it happens that "pkg" is found in a folder that is 
> given in sys.path as a relative path

I'd prefer that Python disallowed relative paths in sys.path [1]. But since they're allowed, I think importlib at least could try to resolve relative paths in a copy of sys.path before searching. 

> as of 3.8 the current directory is removed from the search path, 
> so the .pyd is never found

It happens to work prior to 3.8 even though the load uses the flag LOAD_WITH_ALTERED_SEARCH_PATH, for which it's documented that "[i]f this value is used and lpFileName specifies a relative path, the behavior is undefined". 

The implemented behavior with LOAD_WITH_ALTERED_SEARCH_PATH is that the directory of the given DLL filename is added to the head of the DLL search path, even though it's a relative path. Then the DLL filename is searched for like any other relative filename. 

For example, loading r"foo\spam.pyd" will try to load r"foo\foo\spam.pyd" (note the double "foo"), r"C:\Windows\System32\foo\spam.pyd", r"C:\Windows\System\foo\spam.pyd", r"C:\Windows\foo\spam.pyd", and so on. If the current working directory (i.e. ".") is in the DLL search path, and r"foo\spam.pyd" isn't accidentally found relative to a directory that's searched before ".", then the loader will find r".\foo\spam.pyd". Fortunately another thread can't change the working directory while the loader is searching, since the PEB lock is held. If r"foo\spam.pyd" is found and it depends on "eggs.dll", the loader will look for it first in the DLL directory, i.e. as r"foo\eggs.dll".

The implicit inclusion of the working directory can be disabled or replaced with another directory via SetDllDirectoryW(), in which case the working directory will only be checked if %PATH% contains a "." entry. If it's replaced with another directory, then it's even inheritable from a SetDllDirectoryW() call in an ancestor process.

3.8+ uses the flag LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR, which requires the DLL filename to be a fully-qualified path.

---
[1] That includes the "" entry in sys.path in the interactive shell. I wish it was implemented to resolve the working directory at startup instead of letting the entry vary with the current working directory.
msg387056 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-15 20:32
> since they're allowed, I think importlib at least could try to resolve relative paths in a copy of sys.path before searching. 

I agree with this fix. They can be resolved for each new import, if we think that's an important behaviour to preserve (might mess with the cache... but probably necessary if it's to be backported). But at some point sys.path entries need to be made absolute, and it definitely needs to happen before we try to load any DLLs.
msg389930 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-03-31 19:37
Added one possible change as a PR, but will need the importlib folk to weigh in on whether it's the best place.

I also need to write a test (and possibly fix any tests that currently check that relative paths are _not_ resolved... letting CI handle that for me)
msg390388 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-04-07 00:02
New changeset 04732ca993fa077a8b9640cc77fb2f152339585a by Steve Dower in branch 'master':
bpo-43105: Importlib now resolves relative paths when creating module spec objects from file locations (GH-25121)
https://github.com/python/cpython/commit/04732ca993fa077a8b9640cc77fb2f152339585a
msg390421 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-04-07 11:36
New changeset 0af99b44edf559305def22b2d68be685ce50d7f6 by Steve Dower in branch '3.9':
bpo-43105: Importlib now resolves relative paths when creating module spec objects from file locations (GH-25121)
https://github.com/python/cpython/commit/0af99b44edf559305def22b2d68be685ce50d7f6
msg390423 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-04-07 11:36
Just needs the 3.8 backport - will get to that later tonight.
msg390445 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-04-07 16:29
The 3.8 backport is much more complicated, as we don't have access to the PathSkipRoot function there. So we can't use the native function. There's probably another way to implement the fix for 3.8, but I'm leaving that for another day. Feel free to chime in with suggestions and/or PRs.
msg390476 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-04-07 21:13
> The 3.8 backport is much more complicated, as we don't have access to 
> the PathSkipRoot function there. So we can't use the native function.

I guess you missed the comment that I left on the PR a few days ago. The 3.8 backport can use the older PathSkipRootW() in shlwapi.dll [1]. It works similarly, except it doesn't support a drive relative root such as "C:spam" -> ("C:", "spam"), but it's easy enough to handle that case in C. Also, it's documented that it's limited to MAX_PATH, but it works fine with long paths in Windows 10, even if the process does not have long-path support enabled. Anyway, just limit the copy to the first `MAX_PATH - 1` characters. Practically speaking, no root is anywhere near that long. It would require a ridiculously long device name in a "\\.\" device path. 

Examples:

    import ctypes
    shlwapi = ctypes.WinDLL('shlwapi')
    shlwapi.PathSkipRootW.restype = ctypes.c_wchar_p
    path = (ctypes.c_wchar * 1000)()

It returns NULL if there's no root:

    >>> path.value = r'spam'
    >>> shlwapi.PathSkipRootW(path) is None
    True

Drive-relative paths aren't supported the same as they are in PathCchSkipRoot():

    >>> path.value = r'C:spam'
    >>> shlwapi.PathSkipRootW(path) is None
    True

Otherwise it seems to support the same range of paths as PathCchSkipRoot():

    >>> path.value = r'\spam'
    >>> shlwapi.PathSkipRootW(path)
    'spam'

    >>> path.value = r'C:\spam'
    >>> shlwapi.PathSkipRootW(path)
    'spam'

    >>> path.value = r'\\server\share\spam'
    >>> shlwapi.PathSkipRootW(path)
    'spam'

    >>> path.value = r'\\?\C:\spam'
    >>> shlwapi.PathSkipRootW(path)
    'spam'

    >>> path.value = r'\\?\UNC\server\share\spam'
    >>> shlwapi.PathSkipRootW(path)
    'spam'

    >>> path.value = r'\\?\Volume{12345678-1234-1234-1234-123456789ABC}\spam'
    >>> shlwapi.PathSkipRootW(path)
    'spam'

    >>> path.value = r'\\.\PIPE\spam'
    >>> shlwapi.PathSkipRootW(path)
    'spam'

---

[1] https://docs.microsoft.com/en-us/windows/win32/api/shlwapi/nf-shlwapi-pathskiprootw
msg390672 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-04-09 22:06
New changeset eed7686e9fe22a3eb5e1a1fc7d27c27fca070bd1 by Steve Dower in branch '3.8':
bpo-43105: Importlib now resolves relative paths when creating module spec objects from file locations (GH-25121)
https://github.com/python/cpython/commit/eed7686e9fe22a3eb5e1a1fc7d27c27fca070bd1
msg393129 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-05-06 18:39
This caused a regression described in https://bugs.python.org/issue44061
msg395032 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2021-06-03 18:50
There is a report about this change might have caused behaviour change  for '.' in sys.path between 3.10.0a7 and 3.10.0b1

https://mail.python.org/archives/list/python-dev@python.org/thread/DE3MDGB2JGOJ3X4NWEGJS26BK6PJUPKW/
msg395775 - (view) Author: neonene (neonene) * Date: 2021-06-14 06:28
After this contribution, when using module at the root dir (maybe bad manners), the followings are expected behaviors?

(1) relative drive in sys.path -> bytecode is not put in __pycache__ folder.

>>> import sys
>>> sys.path.append('F:')  # flash device, etc...
>>> import foo
>>> foo.__file__
'F:foo.py'
>>> foo.__cached__
'F:foo.cpython-311.pyc'


(2) absolute drive in sys.path -> __pycache__ is under current dir, not absolute.

>>> import sys
>>> sys.path.append('F:\\')
>>> import foo
>>> foo.__file__
'F:\\foo.py'
>>> foo.__cached__
'F:__pycache__\\foo.cpython-311.pyc'
History
Date User Action Args
2021-06-14 06:28:27neonenesetnosy: + neonene
messages: + msg395775
2021-06-03 18:50:04orsenthilsetnosy: + orsenthil
messages: + msg395032
2021-05-06 18:39:26gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg393129
2021-04-12 15:36:46steve.dowersetstatus: open -> closed
assignee: steve.dower
resolution: fixed
stage: patch review -> resolved
2021-04-09 22:06:27steve.dowersetmessages: + msg390672
2021-04-09 19:38:32steve.dowersetstage: backport needed -> patch review
pull_requests: + pull_request24053
2021-04-07 21:13:43eryksunsetmessages: + msg390476
2021-04-07 16:29:14steve.dowersetmessages: + msg390445
versions: - Python 3.9, Python 3.10
2021-04-07 11:36:42steve.dowersetmessages: + msg390423
stage: patch review -> backport needed
2021-04-07 11:36:00steve.dowersetmessages: + msg390421
2021-04-07 00:30:40steve.dowersetpull_requests: + pull_request23975
2021-04-07 00:02:10steve.dowersetmessages: + msg390388
2021-03-31 19:37:06steve.dowersetmessages: + msg389930
2021-03-31 19:36:16steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request23867
2021-03-30 18:36:51eryksunsettitle: Can't import extension modules resolved via relative paths in sys.path on Windows -> [Windows] Can't import extension modules resolved via relative paths in sys.path
components: + Interpreter Core, Library (Lib), - Extension Modules
versions: - Python 3.7
2021-02-15 20:32:25steve.dowersetmessages: + msg387056
2021-02-13 03:17:36eryksunsetversions: + Python 3.7
2021-02-13 03:14:21eryksunsetnosy: + eryksun, brett.cannon
messages: + msg386898
2021-02-12 22:47:21steve.dowersetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2021-02-02 17:04:14smundaysettitle: Can't import extension modules resolved via relative paths in sys.path on Windows don't don't -> Can't import extension modules resolved via relative paths in sys.path on Windows
2021-02-02 17:03:00smundaycreate