classification
Title: os.dup() fails for standard streams on Windows 7
Type: behavior Stage:
Components: Extension Modules, Windows Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Jeffrey.Kintscher, ZackerySpytz, daveb, eryksun, josh.r, lukasz.langa, ned.deily, paul.moore, steve.dower, tim.golden, vstinner, xflr6, zach.ware
Priority: release blocker Keywords:

Created on 2019-07-10 17:52 by daveb, last changed 2019-07-21 10:32 by xflr6.

Messages (6)
msg347627 - (view) Author: DaveB (daveb) Date: 2019-07-10 17:52
os.dup(fd) generates an OSError for standard streams (0: stdin, 1: stdout, 2: stderr) on Windows 7.  Works as expected on Windows 10.

Working backwards we found the issue first appears in Python 3.7.4rc1 and 3.8.0b2 on Windows 7.  Earlier releases work as expected.

>>> import os
>>> os.dup(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [WinError 87] The parameter is incorrect
>>>
msg347632 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-07-10 18:38
This seems likely to have been caused by the fixes for #37267, which fixes an issue with os.dup leaving character streams inheritable (when the documentation specifies that the result must be non-inheritable).

The code originally didn't try to make the descriptor non-inheritable because someone believed it wasn't allowed for character files, and the subsequent patch comments say "That was a mistake". Is it possible it wasn't allowed on Windows 7, and is allowed on Windows 10?

I'm nosying the folks from #37267 for input.
msg347642 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-07-10 23:46
> OSError: [WinError 87] The parameter is incorrect

This error didn't originate from the C dup() call, since that would be based on errno, like `OSError: [Errno 9] Bad file descriptor`. It must come from _Py_set_inheritable. The only WINAPI call there is SetHandleInformation [1], which is documented to support "console input buffer" and "console screen buffer" handles, though it may be that the documentation is wrong.

Try the following code:

    >>> import os, msvcrt
    >>> msvcrt.get_osfhandle(0)
    84
    >>> os.set_handle_inheritable(msvcrt.get_osfhandle(0), False)

Prior to Windows 8, a console handle is tagged by setting the lower 2 bits (e.g. 3, 7, 11). The system uses this tag to direct calls to special console functions that route requests over the console LPC port. Thus, in the domain of File handles, setting the lower 2 bits is reserved to tag console handles. This should also be true in Windows 8+, even though console handle tagging is no longer used (now they're kernel handles for the ConDrv device). 

The above test will confirm my suspicion, but it looks some time around Windows XP/2003, Microsoft removed the internal [Get|Set]ConsoleHandleInformation functions that used to implement [Get|Set]HandleInformation for console handles, and the people responsible for the change failed to update the docs. Since the internal DuplicateConsoleHandle function wasn't removed, dup() itself still succeeds.

I suggest that _Py_set_inheritable should handle this case. If the call fails with ERROR_INVALID_PARAMETER, and the two tag bits of the handle are set, the handle is possibly (not necessarily) a console handle. In this case, if GetFileType is FILE_TYPE_CHARACTER, then it's a console handle, and the error should be ignored.

[1] https://docs.microsoft.com/en-us/windows/win32/api/handleapi/nf-handleapi-sethandleinformation
msg347644 - (view) Author: DaveB (daveb) Date: 2019-07-10 23:57
Results with Python 3.7.4 on Windows 7

>>> import os, msvcrt
>>> msvcrt.get_osfhandle(0)
15
>>> os.set_handle_inheritable(msvcrt.get_osfhandle(0), False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [WinError 87] The parameter is incorrect
>>>
msg347668 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-11 10:11
Oh, os.dup() doc says: "On Windows, when duplicating a standard stream (0: stdin, 1: stdout, 2: stderr), the new file descriptor is inheritable."
https://docs.python.org/dev/library/os.html

Maybe we should add an inheritable paramater to os.dup() in Python 3.8?
msg347671 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-07-11 11:02
> os.dup() doc says: "On Windows, when duplicating a standard stream 
> (0: stdin, 1: stdout, 2: stderr), the new file descriptor is 
> inheritable."

That's not necessarily correct. For example, if stdout is a pipe or disk file:

    C:\>python -V
    Python 3.7.3
    
    C:\>python -c "import os; fd = os.dup(1); print(os.get_inheritable(fd))" | more
    False

    C:\>python -c "import os; fd = os.dup(1); print(os.get_inheritable(fd))" > stdout.txt

    C:\>type stdout.txt
    False

What 3.7.3 does is to skip calling _Py_set_inheritable for all files of type FILE_TYPE_CHAR, which is wrong for character devices in general, such as NUL and serial ports. NUL is a common target for standard I/O redirection.
History
Date User Action Args
2019-07-21 10:32:08xflr6setnosy: + xflr6
2019-07-14 09:08:25vstinnersetpriority: normal -> release blocker
nosy: + ned.deily, lukasz.langa
2019-07-12 20:05:01Jeffrey.Kintschersetnosy: + Jeffrey.Kintscher
2019-07-11 11:02:44eryksunsetmessages: + msg347671
2019-07-11 10:11:42vstinnersetmessages: + msg347668
2019-07-10 23:57:40davebsetmessages: + msg347644
2019-07-10 23:46:02eryksunsetmessages: + msg347642
2019-07-10 18:38:20josh.rsetnosy: + ZackerySpytz, vstinner, josh.r
messages: + msg347632
2019-07-10 18:19:17xtreaksetnosy: + eryksun
2019-07-10 18:02:57xtreaksetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2019-07-10 17:52:05davebcreate