classification
Title: Py_Initialize Hangs on Windows 10
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: dhamilton, duaneg, paul.moore, ph.fieschi, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2020-01-15 18:55 by dhamilton, last changed 2021-06-09 02:44 by duaneg.

Files
File name Uploaded Description Edit
dup-hang.c duaneg, 2021-06-09 02:44
Messages (4)
msg360070 - (view) Author: Darren Hamilton (dhamilton) Date: 2020-01-15 18:55
This is related to https://bugs.python.org/issue17797, which is closed.

Using Python 3.7.4, Windows 10.0.18362, Visual Studio 2017 and running as a C Application.  Py_Initialize() eventually calls is_valid_fd with STDIN.  The behavior appears to cause both dup() and fstat() to hang indefinitely (using RELEASE MSVCRT DLLs, it works correctly using MSVCRT Debug DLLs).  The call stack shows Windows is waiting for some Windows Event.  The recommended patch in issue17797 will not work.

is_valid_fd appears to want to read the 'input' using a file descriptor.  since both dup and fstat hang, I realized that isatty() would indicate if the file descriptor is valid and works for any predefined FD descriptor(STDIN-0, STDOUT-1, STDERR-2).

#if defined(MS_WINDOWS)
	struct stat buf;
	if (fd >= fileno(stdin) && fd <= fileno(stderr)) {
		return (_isatty(fd) == 0 && errno == EBADF) ? 0 : 1;
	}
	else if (fstat(fd, &buf) < 0 && (errno == EBADF || errno == ENOENT))
		return 0;
	return 1;
#else
msg375628 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-18 21:50
Are you able to capture a process dump at the hang? I haven't seen this anywhere else, and don't even know how to go about trying to reproduce it with this information - Py_Initialize is called by every single Python process, so there's something special about your situation that isn't obvious yet :)
msg375639 - (view) Author: Philémon Fieschi (ph.fieschi) Date: 2020-08-19 09:00
Personally, I have the same problem of Py_Initialize() hanging indefinitely.

Here is the context in which it happens : I am developing an application in Java, in which I use the library jep (https://github.com/ninia/jep), that enables me to get a Python interpreter from Java, and I am developing and testing it on Windows. My Python version is 3.8.2 and I am on Windows 10 - version 1903.
When I test this library outside my app in a simple Java project, everything works fine, and the interpreter works. 
But when I try to use it in the app, it hangs indefinitely when I create the interpreter. When I digged into the code of the library, I found out that it occurs in the native code of jep, during the call to Py_Initialize(). I posted an issue on the github of jep, and they brought me here. I bet this is related to stdin and stdout when I see what dhamilton posted.
My Java's stdout is normal and writes in the console. I tried to reset or redirect Java's stdin an stdout, but it doesn't change nothing.

And when I try to do this on Linux (my application is also on Linux), on Ubuntu 16, everything works fine and it doesn't hang indefinitely. So this only happens on Windows.

About capturing a process dump, all I can get is a message displayed on the Java console when I close the app (because it hangs indefinitely) :
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000302b9d8f, pid=11960, tid=0x0000000000003f98
#
# JRE version: Java(TM) SE Runtime Environment (8.0_241-b07) (build 1.8.0_241-b07)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.241-b07 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  0x00000000302b9d8f
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows

# The crash happened outside the Java Virtual Machine in native code.

All of this is just my personal case, and it's probably not the same for dhamilton. I hope it helped you.
msg395377 - (view) Author: Duane Griffin (duaneg) * Date: 2021-06-09 02:44
I can reproduce this on Windows 10 with Python 3.9. See attached source. At least for us, it is hanging when one thread is doing a read on the file descriptor while a second calls Py_Initialize (or just dup directly).

The windows kernel call stack shows the dup call is waiting on a critical section, while the thread reading from stdin is waiting in ReadFile. I can get a full stack trace from WinDbg if it is helpful, but hopefully the attached code should be enough to reproduce the problem at will for anyone interested.

If stdin is receiving input, or is closed, then the read call will complete and unblock dup in due course. However if not then it will hang indefinitely.

If we can fix this to work reliably in Python that would be great. Otherwise, or in the meantime, we could just add a note to the documentation. We are going to try and work-around it by using a different file descriptor instead of stdin. Other applications might be able to avoid IO using stdin until after python is initialised.
History
Date User Action Args
2021-06-09 02:44:42duanegsetfiles: + dup-hang.c
nosy: + duaneg
messages: + msg395377

2020-08-19 09:00:03ph.fieschisetmessages: + msg375639
2020-08-18 21:50:43steve.dowersetmessages: + msg375628
2020-08-18 13:47:45ph.fieschisetnosy: + ph.fieschi
2020-01-15 18:55:56dhamiltoncreate