Title: failed pickle hangs python on exit in CMD.exe only
Type: crash Stage: patch review
Components: Windows Versions: Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: John-Ted, charles.mcmarrow.4, davin, eryksun, paul.moore, pitrou, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2020-10-27 20:52 by charles.mcmarrow.4, last changed 2020-11-16 18:29 by eryksun.

File name Uploaded Description Edit charles.mcmarrow.4, 2020-10-27 20:52 run in cmd to get hang
Pull Requests
URL Status Linked Edit
PR 23290 open John-Ted, 2020-11-14 21:51
Messages (3)
msg379803 - (view) Author: Charles McMarrow (charles.mcmarrow.4) Date: 2020-10-27 20:52
This bug only happens for me in Windows 10 when using cmd.exe. This bug does not happen IDLE. It sometimes happens in PowerShell. 

I ran
3.9.0 hangs on cmd/powershell
3.8.6 hangs on cmd
3.8.5 hangs on cmd/powershell
Microsoft Windows [Version 10.0.19041.572]

3.8.6 error
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\yoshi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\", line 102, in spawn_main
    source_process = _winapi.OpenProcess(
OSError: [WinError 87] The parameter is incorrect
msg379816 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-28 03:36
Due to the exception raised when trying to pickle main.<locals>.dummy in popen_spawn_win32.Popen(), the main process may have already exited by the time the spawned worker calls WinAPI OpenProcess in spawn.spawn_main(). I suppose the reduction of prep_data and process_obj  in popen_spawn_win32.Popen() could be dumped to an io.BytesIO instance before WinAPI CreateProcess is called. That way an exception in the pickler would occur before the child process is created.

There's also a race condition that can hang the worker during interpreter startup. When sys.stdin is initialized, the buffered reader does a raw tell(). If the raw layer is an io.FileIO instance (e.g. console input with PYTHONLEGACYWINDOWSSTDIO defined), the raw tell() ultimately calls WinAPI SetFilePointerEx (via _lseeki64). This can hang if the kernel file is already blocked in a synchronous I/O request, which might be the case if the main process has already terminated and its parent (e.g. cmd.exe or powershell.exe) has resumed reading from the console. portable_lseek() in Modules/_io/fileio.c should detect a non-disk file via WinAPI GetFileType and just set self->seekable to 0 in this case.
msg381138 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-11-16 18:29
Steve or Zach, please review PR 23290, submitted by Teugea Ioan-Teodor. It modifies the startup sequence for worker processes by creating the child process with a suspended thread and pickling process_obj to a BytesIO instance before resuming the thread and writing the pickle data to the pipe. If pickling fails, the child process is terminated before the thread is resumed, since there's no point in allowing the child to execute. 

This doesn't directly address the issue with interpreter startup that can cause Python to hang on a synchronous I/O request, but it does avoid the problem.

Alternatively, when there's a pickling error, the parent could close its end of the pipe and wait for the child to exit before propagating the exception, but I'd rather skip executing the child completely.
Date User Action Args
2020-11-16 18:29:48eryksunsetmessages: + msg381138
2020-11-16 15:27:15eryksunsetnosy: + pitrou, davin
2020-11-14 21:51:35John-Tedsetkeywords: + patch
nosy: + John-Ted

pull_requests: + pull_request22181
stage: patch review
2020-10-28 03:36:26eryksunsetnosy: + eryksun
messages: + msg379816
2020-10-27 20:52:33charles.mcmarrow.4create