This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: subprocess.check_output() fails with OSError: [WinError 87] when current directory name is too long
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.10, Python 3.9, Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Geoff.Alexander, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-03-06 14:37 by Geoff.Alexander, last changed 2022-04-11 14:59 by admin.

Messages (7)
msg337307 - (view) Author: Geoff Alexander (Geoff.Alexander) Date: 2019-03-06 14:37
I've found that subprocess.check_output() fails on Windows with OSError: [WinError 87] when the current directory's name is too long:

Traceback (most recent call last):
  File "", line 169, in <module>
  File "", line 80, in migrate
    rtc.acceptchangesintoworkspace(rtc.getchangeentriestoaccept(changeentries, history))
  File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\", line 310, in acceptchangesintoworkspace
  File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\", line 97, in addandcommit
  File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\", line 130, in handle_captitalization_filename_changes
    files = shell.getoutput("git ls-files")
  File "c:\Users\GeoffAlexander\Documents\Nirvana\RTC2Git\git-repositories\rtc2git-migration-tool\", line 33, in getoutput
    outputasbytestring = check_output(command, shell=True)
  File "C:\Users\GeoffAlexander\AppData\Local\Programs\Python\Python36\lib\", line 356, in check_output
  File "C:\Users\GeoffAlexander\AppData\Local\Programs\Python\Python36\lib\", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\GeoffAlexander\AppData\Local\Programs\Python\Python36\lib\", line 729, in __init__
    restore_signals, start_new_session)
  File "C:\Users\GeoffAlexander\AppData\Local\Programs\Python\Python36\lib\", line 1017, in _execute_child
OSError: [WinError 87] The parameter is incorrect

Python's subprocess module should handle long directory and files names on Windows where supported.  For older versions of Windows that don't support long directory and file names, an exception with a more informative error message than "OSError: [WinError 87]" should be thrown.
msg337309 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-06 14:49
subprocess.Popen calls _winapi.CreateProcess. The current_directory is a wchar_t* string (Py_UNICODE*) passed to Windows CreateProcessW() (Unicode flavor of the API).

"In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters."

I guess that the workaround is to use the "\\?\" prefix for "extended-length path". For example, "\\?\D:\very long path".

=> "The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters."
msg337314 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-03-06 15:06
The problem with improving the generic error message is that we then build a platform limitation into Python and if a later version of Windows fixes it then Python needs to make another fix.

As Eryk said on the other issue, this may be because the current working directory of the child process can't be set to a long path. The extended syntax may not help here, though it's easy enough to test. Geoff - can you try adding \\?\ (in Python, "\\\\?\\") before the cwd and passing it into the call? If that doesn't work then I don't know that there's much we can do.
msg337315 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-06 15:09
> As Eryk said on the other issue, ...

msg337357 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-03-07 03:45
Long-path support in Windows 10 does not extend to the lpCurrentDirectory parameter of CreateProcessW. If the path length exceeds the old limit of MAX_PATH - 2 characters (not counting the required trailing backslash and null), CreateProcessW fails with either ERROR_DIRECTORY (267) or ERROR_INVALID_PARAMETER (87).

If we pass lpCurrentDirectory as a long path, it fails with ERROR_DIRECTORY (i.e. the directory name is invalid). This is a bit confusing because it maps to Python NotADirectoryError, which is the common usage for ERROR_DIRECTORY. CreateProcessW, however, uses it in its broadest sense. It also fails with this error if lpCurrentDirectory can't be validated as an existing directory via GetFileAttributesW.

If we pass lpCurrentDirectory as NULL (i.e. inherit the current directory) and our current directory is a long path, CreateProcessW fails with ERROR_INVALID_PARAMETER. The source of this error isn't obvious unless we know what to look for, since all of the passed parameters are in fact valid. However, I'm not sure how to clarify the error without making assumptions. In particular, a future release of Windows 10 may remove this limitation, in which case we could end up obscuring an unrelated error. 

> the workaround is to use the "\\?\" prefix

Without long-path support, the working directory is always limited to MAX_PATH - 2 characters. The \\?\ prefix doesn't help. A few years ago, the documentation for SetCurrentDirectory [1] was changed to include an invalid claim that we can use the \\?\ prefix. We need to go back to 2016 [2] to get the correct documentation. 

Windows doesn't even fully support setting the current directory to a device path (i.e. prefixed by \\.\ or \\?\). Operations on rooted paths will fail badly because the system computes an invalid working drive in this case. We can observe the problem by calling GetFullPathNameW:

    >>> os.chdir(r'\\.\C:\Temp')
    >>> print(os.path._getfullpathname(r'\Temp'))

The incorrect result might even be a valid path, coincidentally. For example:

    >>> print(os.getcwd())
    >>> os.chdir(r'\localhost\C$')
    >>> print(os.getcwd())

msg337522 - (view) Author: Geoff Alexander (Geoff.Alexander) Date: 2019-03-08 18:18
Using the "\\?\" prefix does not work.  Here's a small example:

import os
import subprocess

output = subprocess.check_output("dir", shell=True)

Using Python 3.7.2 64-bit on Windows 10 fails with

CMD.EXE was started with the above path as the current directory.
UNC paths are not supported.  Defaulting to Windows directory.
msg388811 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-16 04:31
As discussed in msg337357, the Windows API does not support setting the current working directory to a path that starts with \\?\ or \\.\. It's dysfunctional in many cases. Anyway, using a \\?\ path does not remove the length limit on the working directory, which is a hard limit in a process that does not support long DOS paths. I assume this is why CreateProcessW() hasn't been extended to support long paths in lpCurrentDirectory. 

The only behavior that can be addressed here is what to do when CreateProcessW() fails with ERROR_INVALID_PARAMETER (87) if the working directory of the current process is a long path. Maybe subprocess should emit a warning for this error if the working directory is a long path, at least to hint to a developer that this could be the reason for the error. Otherwise the error is difficult to understand if you don't already know what to look for.
Date User Action Args
2022-04-11 14:59:12adminsetgithub: 80394
2021-03-16 10:48:05vstinnersetnosy: - vstinner
2021-03-16 04:31:38eryksunsetmessages: + msg388811
components: + Library (Lib)
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 3.6, Python 3.7
2019-03-08 18:18:18Geoff.Alexandersetmessages: + msg337522
2019-03-07 03:45:46eryksunsetnosy: + eryksun
messages: + msg337357
2019-03-06 15:09:48vstinnersetmessages: + msg337315
2019-03-06 15:06:28steve.dowersetmessages: + msg337314
2019-03-06 14:49:45vstinnersetnosy: + vstinner
messages: + msg337309
2019-03-06 14:44:40SilentGhostsettype: behavior
versions: + Python 3.6
2019-03-06 14:37:16Geoff.Alexandercreate