Opening a file calls isatty which calls an ioctl with the GIL held.

GDB:

#0  __GI___tcgetattr (fd=18, termios_p=termios_p@entry=0x7f618a5df920)
    at ../sysdeps/unix/sysv/linux/tcgetattr.c:38
#1  0x00007f618bd1ca0c in __isatty (fd=<optimized out>) at ../sysdeps/posix/isatty.c:27
#2  0x000000000062b746 in _Py_device_encoding (fd=<optimized out>) at ../Python/fileutils.c:62
#3  0x000000000060bf90 in _io_TextIOWrapper___init___impl (write_through=0, line_buffering=0, 
    newline=0x0, errors='strict', encoding=<optimized out>, 
    buffer=<_io.BufferedWriter at remote 0x7f618986aeb0>, self=0x7f618985a860)
    at ../Modules/_io/textio.c:1149
...

Please don't do that.

In my case, the file in question is implemented as a FUSE mount which is served by the same process (different thread of course). Thus holding the GIL at this point causes a rather interesting deadlock.

Tested with 3.9.

My team ran into this issue while developing a fuse application too.

In an effort to help this issue move forward, I tried to list all occurrences of the isatty C function in the cpython code base. I found 14 of them.

9 of them are directly related to stdin/stdout/stderr, so it's probably not crucial to release the GIL for those occurrences:

• main.c:stdin_is_interactive
• main.c:pymain_import_readline
• readline.c:setup_readline
• bltinmodule.c:builtin_input_impl (x2)
• frozenmain.c:Py_FrozenMain
• pylifecycle.c:Py_FdIsInteractive (x2)
• fileobject.c:stdprinter_isatty (GIL is actually released for this one)

Out of the remaining 4, only 1 releases the GIL:

• fileio.c:_io_FileIO_isatty_impl: used for FileIO.isatty

Which gives 3 occurrences of non-stdstream specific usage of isatty that do not release the GIL:

• posixmodule.c:os_isatty_impl: used by os.isatty
• fileutils.c:_Py_device_encoding: used TextIOWrapper.__init__
• fileutils.c:_Py_write_impl: windows specific, issue bpo-11395

The first one is used by os.isatty which means this call can also deadlock. I did manage to reproduce it with a simple fuse loopback file system: https://github.com/fusepy/fusepy/blob/master/examples/loopback.py

The second one is the one found by @smurfix and gets triggered when io.open() is used in text mode.

The third one only triggers on windows when writing more than 32767 bytes to a file descriptor. A comment points to issue bpo-11395 (https://bugs.python.org/issue11395). Also, it seems from the function signature that this function might be called with or without the GIL held, which might cause the fix to be a bit more complicated than the first two use cases.

I hope this helps.

Here's a possible patch that fixes the 3 unprotected calls to isatty mentioned above. It successfully passes the test suite. I can submit a PR with this patch if necessary.

Please do. However, I do think that changing the stdstream related ioctl calls also is a good idea, if only for code regularity/completeness sake.

(Besides, nothing prevents somebody from starting a FUSE file system and then redirecting stdout to it …)

There are a couple of reasons why I did not make changes to the stdstream related functions.

The first one is that a PR with many changes is less likely to get reviewed and merged than a PR with fewer changes. The second one is that it's hard for me to make sure that those functions are always called with the GIL already held. Maybe some of them never hold the GIL in the first place, and I'm not familiar enough with the code base to tell.

So in the end it will probably be up to the coredev reviewing the PR, but better start small.

Besides, nothing prevents somebody from starting a FUSE file system and then redirecting stdout to it …

I ran some checks and python -c 'input()' < my-fuse-mountpoint/data_in does indeed trigger an ioctl call to the corresponding fuse file system. But how would my-fuse-mountpoint/data_in be the stdin for the process that itself starts the fuse filesystem? I don't see how to generate this deadlock, apart from setting up interprocess communication between the fuse process and the my-fuse-mountpoint/data_in-as-stdin process.

New changeset 06148b1 by Vincent Michel in branch 'main':
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
06148b1

New changeset 5c65834 by Miss Islington (bot) in branch '3.9':
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
5c65834

New changeset 23c4677 by Miss Islington (bot) in branch '3.10':
bpo-44219: Release the GIL during isatty syscalls (GH-28250) (GH-28255)
23c4677

In my case, the file in question is implemented as a FUSE mount which is served by the same process (different thread of course). Thus holding the GIL at this point causes a rather interesting deadlock.

Since the change fixes a deadlock, I agree to backport it to 3.9 and 3.10.

Thanks for the fix!

New changeset 296b710 by Łukasz Langa in branch 'main':
bpo-44219: Mention #72437 is a deadlock bugfix (GH-28261)
296b710

New changeset 60ddf49 by Miss Islington (bot) in branch '3.10':
bpo-44219: Mention #72437 is a deadlock bugfix (GH-28261) (GH-28274)
60ddf49

New changeset 314de53 by Miss Islington (bot) in branch '3.9':
bpo-44219: Mention #72437 is a deadlock bugfix (GH-28261) (GH-28275)
314de53