This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: The Python process stops responding immediately after starting
Type: behavior Stage: resolved
Components: Interpreter Core, Windows Versions: Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: [Windows] Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows
View: 34780
Assigned To: Nosy List: eryksun, malin, paul.moore, steve.dower, tim.golden, zach.ware, Александр Виноградов
Priority: normal Keywords:

Created on 2016-04-29 08:12 by Александр Виноградов, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
python.dmp Александр Виноградов, 2016-04-29 08:12 Dump of freezed process
Messages (4)
msg264475 - (view) Author: Александр Виноградов (Александр Виноградов) Date: 2016-04-29 08:12
I start in Windows 7 virtual machine the Python x86 subprocess from another console application with commandline: 
c:\python35\python.exe -c print('hello')

Immediately after the startup process stops responding and hanging forever.

If you run it with the parameter -version python shows the version information to the console normally.
msg264486 - (view) Author: Ma Lin (malin) * Date: 2016-04-29 10:43
Hang everytime?
When it hangs, try to press ENTER of keyboard on the console window, does it work?
If it works, see issue26744: print() function hangs on MS-Windows 10
msg264499 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-04-29 14:11
python.dmp is from Windows 8.1 (NT 6.3):

    0:000> ?? @$peb->OSMajorVersion; ?? @$peb->OSMinorVersion
    unsigned long 6
    unsigned long 3

with the following command line and working directory:

    0:000> ?? @$peb->ProcessParameters->CommandLine
    struct _UNICODE_STRING
     ""C:\Python35\python.exe" -c print('hello')"
       +0x000 Length           : 0x54
       +0x002 MaximumLength    : 0x56
       +0x004 Buffer           : 0x01100a9e  ""C:\Python35\python.exe" -c print('hello')"

    0:000> ?? @$peb->ProcessParameters->CurrentDirectory
    struct _CURDIR
       +0x000 DosPath          : _UNICODE_STRING "C:\TestPlatform\integrationTests\"
       +0x008 Handle           : 0x00000024 Void

The STARTUPINFO dwFlags is set to STARTF_USESTDHANDLES (0x100), so the standard handles are from the STARTUPINFO hStdInput, hStdOutput, and hStdError:

    0:000> ?? @$peb->ProcessParameters->WindowFlags
    unsigned long 0x100

Standard input appears to be a console handle (starting in Windows 8 console handles are kernel handles, so we can inspect them in the debugger):

    0:000> ?? @$peb->ProcessParameters->StandardInput
    void * 0x00000018
    0:000> !handle 18 3
    Handle 00000018
      Type          File
      Attributes    0
      GrantedAccess 0x12019f:
             ReadControl,Synch
             Read/List,Write/Add,Append/SubDir/CreatePipe,ReadEA,WriteEA,ReadAttr,WriteAttr
      HandleCount   4
      PointerCount  131064

It looks like a console handle because it's open with both read and write access, which is required for the console API (e.g. ReadConsoleInput and WriteConsoleInput). 

Standard output appears to be a pipe or file, opened with only write access:

    0:000> ?? @$peb->ProcessParameters->StandardOutput
    void * 0x000011d4
    0:000> !handle 11d4 3
    Handle 000011d4
      Type          File
      Attributes    0
      GrantedAccess 0x120196:
             ReadControl,Synch
             Write/Add,Append/SubDir/CreatePipe,WriteEA,ReadAttr,WriteAttr
      HandleCount   2
      PointerCount  65536

The process is blocked on the NtQueryInformationFile [1] system call in the process of creating Python's sys.std* file objects.

    0:000> k
    ChildEBP RetAddr
    0109f354 74afa9a4 ntdll!NtQueryInformationFile+0xc
    0109f3ac 73e11b15 KERNELBASE!SetFilePointerEx+0x9c
    ...
    0109f450 73fe051e ucrtbase!_lseeki64+0x19
    ...
    0109f48c 740421e3 python35!_io_FileIO_tell+0x2d
    ...
    0109f508 73fe4948 python35!_buffered_raw_tell+0x20
    ...
    0109f53c 74053dc0 python35!_io_BufferedReader___init__+0x3f
    ...
    0109f648 740420ff python35!_io_open+0x90
    ...
    0109f710 740c37b4 python35!create_stdio+0xac
    0109f748 740c2871 python35!initstdio+0x264
    0109f760 73faf203 python35!_Py_InitializeEx_Private+0x351
    ...
    0109f808 1c7c11df python35!Py_Main+0x713

Specifically it's querying StandardInput (handle 0x18) for FilePositionInformation (0xe), which is stored in an 8-byte FILE_POSITION_INFORMATION struct (i.e. the size of a LARGE_INTEGER for the current position):

    0:000> dd @esp l6
    0109f358  74afa9a4 00000018 0109f37c 0109f388
    0109f368  00000008 0000000e

This is querying the value of the file pointer to initialize sys.stdin.buffer. 

Microsoft has been busy rolling out major updates to the console subsystem over the past few years. It could be a bug in the condrv.sys console device driver, which is new in Windows 8. It's hard to say what's going on without attaching a kernel debugger to inspect the problem.

Switching to a device driver required reimplementing the console API -- e.g. everything that used to get routed to LPC interprocess calls to conhost.exe now goes through I/O system calls. Specifically, console handles prior to Windows 8 were flagged by setting the lower 2 bits (e.g. 3, 7, 11, etc) to route calls to LPC-based functions where possible (e.g. ReadFile -> ReadConsoleA) or immediately fail. Most if not all of that specialized code is gone in Windows 8+. WinAPI calls with console handles now take the regular path to call NT I/O system calls such as NtReadFile, NtDeviceIoControlFile, and NtQueryInformationFile.

For example, in Windows 7 calling SetFilePointerEx to get or set the non-existent file pointer of a console handle fails immediately with STATUS_INVALID_HANDLE. This can be ignored harmlessly. But in the new console, we can do something silly like this (tested in Windows 10):

    >>> sys.stdin.seek(50)
    50
    >>> sys.stdin.tell()
    50

    >>> kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
    >>> h = kernel32.GetStdHandle(-10)
    >>> pos = ctypes.c_longlong()
    >>> kernel32.SetFilePointerEx(h, 0, ctypes.byref(pos), 1)
    1
    >>> pos.value
    50

(On a tangent, the new implementation also breaks Ctrl+C handling. It no longer sets ERROR_OPERATION_ABORTED when interrupting a ReadFile console read. This breaks Python's REPL and input function.)

[1]: https://msdn.microsoft.com/en-us/library/ff567052
msg388544 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-12 18:43
Hanging on a synchronous console file during startup shouldn't be an issue in 3.6+, since io._WindowsConsoleIO doesn't support seeking, but it could still be an issue with legacy mode, which uses io.FileIO. I'm marking this as a duplicate of bpo-34780, which has more information. It's basically the same problem that seeking should only be supported for files opened for physical disks, volumes, and regular data files in mounted filesystems -- for which its meaningful, useful, and not vulnerable to hanging indefinitely if the file is blocked on a synchronous I/O request (e.g. a read request from a pipe or console input, which might never complete).
History
Date User Action Args
2022-04-11 14:58:30adminsetgithub: 71069
2021-03-12 18:43:40eryksunsetstatus: open -> closed
superseder: [Windows] Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows
messages: + msg388544

resolution: duplicate
stage: resolved
2016-04-29 14:11:55eryksunsetnosy: + eryksun, paul.moore, tim.golden, zach.ware, steve.dower
messages: + msg264499
components: + Windows
2016-04-29 10:43:38malinsetnosy: + malin
messages: + msg264486
2016-04-29 08:12:01Александр Виноградовcreate