classification
Title: code.InteractiveConsole.interact() closes stdin
Type: enhancement Stage: needs patch
Components: Documentation, Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Yonatan Zunger, docs@python, eryksun, terry.reedy
Priority: normal Keywords:

Created on 2018-07-14 22:49 by Yonatan Zunger, last changed 2018-07-24 19:43 by Yonatan Zunger.

Messages (8)
msg321666 - (view) Author: Yonatan Zunger (Yonatan Zunger) Date: 2018-07-14 22:49
code.InteractiveConsole.interact() closes stdin on exit, which can be very surprising to successive code, not least future calls to interact(). A simple repro with a workaround is:

import code
import io
import os
import sys

def run():
    print(sys.stdin.buffer.raw)
    dupstdin = os.dup(0)
    try:
        code.InteractiveConsole().interact()
    except SystemExit:
        pass
    finally:
        # Workaround: Without this line, the second call to run() will fail with a ValueError when
        # it tries to call input().
        sys.stdin = io.TextIOWrapper(
            io.BufferedReader(io.FileIO(dupstdin, mode='rb', closefd=False)),
            encoding='utf8')

run()
run()


- The exciting behavior appears to happen inside the exec() of a 'quit()' command, and I haven't searched it out further.
- That behavior inside exec() is likely there for a good reason, in which case the best fix is probably to just save and restore stdin in the code library.
msg322039 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-07-20 23:40
To investigate your claim about closing sys.stdin, I ran the following on Windows 10, mostly with 3.7.0, in both the console and IDLE, using various exit methods.

import code
import sys

for i in range(2):
    try:
        code.InteractiveConsole().interact()
        print(f'Try {i}: closed is', sys.stdin.closed)
    except SystemExit:
        print(f'Exc {i}: closed is', sys.stdin.closed)

^D in IDLE and ^Z+<enter> in console prints 'Try 0/1...False'.
The IC.interact loop catches EOFError and breaks for a normal exit.

'raise SystemExit' and sys.exit() print 'Exc 0/1...False'.

exit() and quit() print 'Exc 0/1...True' on Console.  On Windows Console, sys.stdin.close() does not prevent a second interact call.  This might be considered a bug.

What OS are you running and what is the result and traceback for the code above?

In IDLE, either function exit causes a 'Kill the running process' popup.  If yes, user code execution ceases and the shell window  closes.  If no, SystemExit is caught and 'SysExit...False' is printed.  The latter is true because sys.stdin.close in the user process is 'sys.shell.close()', and the latter does the popup.  The *purpose* of sys.stdin.close is to get the attention of shells that intercept SystemExit, so that users can say 'Close this now'.  Even so, IDLE asks users first to make sure.

Emulating IDLE in a simplified fashion as follows will negate closure.
import _io
class Stdin(_io.TextIOWrapper):
    def close(self): pass
sys.stdin = Stdin()  # To restore, sys.__stdin__ is the original.


---
History of sys.stdin.close:

https://github.com/python/cpython/commit/24cb053b158a3cd63f7be05ac27f47e45bb2f1b3
Site.py, line 236, Mar 9, 2006, George Brandel added Quitter with __call__ consisting of 'raise SystemExit(code)'.

https://github.com/python/cpython/commit/d112bc7958151fa17c4ccb27413c43e45b8476fb#diff-f34a16518c608b2ca946d3f5ca0a1942
site.py, line Aug 16, 2006, Kurt Kaiser added the following lines
        # Shells like IDLE catch the SystemExit, but listen when their
        # stdin wrapper is closed.
        try:
            sys.stdin.close()
        except:
            pass

https://github.com/python/cpython/commit/862543aa85249b46649b60da96743b4b14c6c83b#diff-f34a16518c608b2ca946d3f5ca0a1942
site.py, line 250, Christian Heimes replace stdin.close with the following to avoid Lib/io.py: RuntimeWarning: Trying to close unclosable fd. (I believe io has since been patched to not do this.)
        fd = -1
        if hasattr(sys.stdin, "fileno"):
            fd = sys.stdin.fileno()
        if fd != 0:
            # Don't close stdin if it wraps fd 0
            sys.stdin.close()

https://hg.python.org/cpython/rev/82451c88b3c0, 11 Apr 2013
After discussion on #17585 with Antoine Pitrou and Serhiy Storchaka, Roger Serwy reverted the above because it was no longer needed 
---

Your code assumes that sys.stdin has a buffer attribute.  Not necessarily true when python is started with pythonw.exe.  With the print removed, it also assumes that sys.stdin has file descriptor 0.  Ditto.
---

Possible doc improvements: the exit/quit doc says that closes.
https://docs.python.org/3/library/constants.html#constants-added-by-the-site-module

interact doc mentions catching exceptions and quit/exit issue.
msg322055 - (view) Author: Yonatan Zunger (Yonatan Zunger) Date: 2018-07-21 01:09
Testing your code sample on OS X (10.13.6) with Python 3.6.2:

- quit() in the console yields

Exc 0: closed is True
Python 3.6.2 (default, Apr 17 2018, 12:29:33) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> Traceback (most recent call last):
  File "./test.py", line 6, in <module>
    code.InteractiveConsole().interact()
  File "/Users/zunger/.pyenv/versions/3.6.2/lib/python3.6/code.py", line 228, in interact
    line = self.raw_input(prompt)
  File "/Users/zunger/.pyenv/versions/3.6.2/lib/python3.6/code.py", line 275, in raw_input
    return input(prompt)
ValueError: I/O operation on closed file.

- ^D and raise SystemExit in console both yield:

Python 3.6.2 (default, Apr 17 2018, 12:29:33) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> ^D
now exiting InteractiveConsole...
Try 0: closed is False
Python 3.6.2 (default, Apr 17 2018, 12:29:33) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> ^D
now exiting InteractiveConsole...
Try 1: closed is False


So it looks like the issue is specific to quit(), as you say. I'm not sure I understand what the purpose of the sys.stdin.close() call is, though: why would we *want* to alert the parent shell in some way beyond a SystemExit as the default (non-overrideable) behavior? This may be a documentation issue, but it seems extremely surprising to me; no other function does this additional thing, even when raising a SystemExit. I would think that any caller which is running InteractiveConsole.interact() and also wants to do something when it finishes would want to handle that additional logic itself.
msg322057 - (view) Author: Yonatan Zunger (Yonatan Zunger) Date: 2018-07-21 01:43
Or perhaps in an alternate phrasing: The sys.stdin.close behavior makes sense if quit is being used inside IDLE, but is very surprising from the perspective of the `code` module. (Really, even the SystemExit is surprising there, and should be documented!)

What if we added a stdin-preserving workaround to code.InteractiveConsole.interact, and added documentation of the SystemExit behavior to that module as well, but left _Quitter itself as-is?
msg322073 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-07-21 05:16
There is an important difference between a program saying 'I am done executing' and a user saying 'I am done with the interactive session'.  This is especially true in an IDE where 'session' can include many editing and shell sessions.

'Stop executing'  happens when execution reaches the end of the file, which causes EOFError upon a read attempt.  It can also be done gracefully before the end of input with sys.exit(), which raises SystemExit.

In Interactive Python, 'leave session' can be done with SystemExit or the EOF control signal, which appears to raise EOFError.  This suggests that quit() and exit(), which were added because newbies did not know the proper way to exit, should raise EOFError rather than SystemExit.  The fact that 'quit' displays 'Use quit() or Ctrl-Z plus Return to exit' (EOF on Windows) suggests the same.  But I need to experiment (another day).
msg322123 - (view) Author: Yonatan Zunger (Yonatan Zunger) Date: 2018-07-21 21:14
Definitely agree about the difference.

I'd say that either SystemExit or EOFError would be a reasonable thing for
the interactive session to do, but the combination of closing stdin and
SystemExit is really weird. Honestly, I would have just expected interact()
to return like an ordinary function when it was done; the logic of "it's
time to close the terminal window" feels like it belongs at a *much* higher
level of the stack.

On Fri, Jul 20, 2018 at 10:16 PM Terry J. Reedy <report@bugs.python.org>
wrote:

>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> There is an important difference between a program saying 'I am done
> executing' and a user saying 'I am done with the interactive session'.
> This is especially true in an IDE where 'session' can include many editing
> and shell sessions.
>
> 'Stop executing'  happens when execution reaches the end of the file,
> which causes EOFError upon a read attempt.  It can also be done gracefully
> before the end of input with sys.exit(), which raises SystemExit.
>
> In Interactive Python, 'leave session' can be done with SystemExit or the
> EOF control signal, which appears to raise EOFError.  This suggests that
> quit() and exit(), which were added because newbies did not know the proper
> way to exit, should raise EOFError rather than SystemExit.  The fact that
> 'quit' displays 'Use quit() or Ctrl-Z plus Return to exit' (EOF on Windows)
> suggests the same.  But I need to experiment (another day).
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue34115>
> _______________________________________
>
msg322139 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-07-22 09:12
> On Windows Console, sys.stdin.close() does not prevent a second 
> interact call.  This might be considered a bug.

This is a bug in io._WindowsConsoleIO. 

In Python 3, the sys.std* file objects that get created at startup use closefd=False:

    >>> sys.stdin.buffer.raw.closefd
    False
    >>> sys.stdout.buffer.raw.closefd
    False
    >>> sys.stderr.buffer.raw.closefd
    False

Since the REPL uses C FILE streams (or in 3.6+ the underlying console file handle in Windows), closing sys.stdin does not cause the REPL to exit, and the PyOS_ReadLine call in the interactive loop continues to work in both POSIX and Windows. 

That said, closing sys.stdin should cause input() to raise ValueError due to sys.stdin.fileno() failing (i.e. take the non-tty path) and subsequently sys.stdin.readline() failing. A second call to code.InteractiveConsole.interact() should thus fail. The issue is that the fileno() method of _WindowsConsoleIO isn't raising ValueError like it should when the file is closed and closefd is false. I've created issue 34187 with a suggested fix.
msg322325 - (view) Author: Yonatan Zunger (Yonatan Zunger) Date: 2018-07-24 19:43
Eryk: Thanks for finding that! So that I'm sure I understand, if 34187 is
resolved, does that mean the stdin.close() is no longer required at all in
_Quitter?

On Sun, Jul 22, 2018 at 2:12 AM Eryk Sun <report@bugs.python.org> wrote:

>
> Eryk Sun <eryksun@gmail.com> added the comment:
>
> > On Windows Console, sys.stdin.close() does not prevent a second
> > interact call.  This might be considered a bug.
>
> This is a bug in io._WindowsConsoleIO.
>
> In Python 3, the sys.std* file objects that get created at startup use
> closefd=False:
>
>     >>> sys.stdin.buffer.raw.closefd
>     False
>     >>> sys.stdout.buffer.raw.closefd
>     False
>     >>> sys.stderr.buffer.raw.closefd
>     False
>
> Since the REPL uses C FILE streams (or in 3.6+ the underlying console file
> handle in Windows), closing sys.stdin does not cause the REPL to exit, and
> the PyOS_ReadLine call in the interactive loop continues to work in both
> POSIX and Windows.
>
> That said, closing sys.stdin should cause input() to raise ValueError due
> to sys.stdin.fileno() failing (i.e. take the non-tty path) and subsequently
> sys.stdin.readline() failing. A second call to
> code.InteractiveConsole.interact() should thus fail. The issue is that the
> fileno() method of _WindowsConsoleIO isn't raising ValueError like it
> should when the file is closed and closefd is false. I've created issue
> 34187 with a suggested fix.
>
> ----------
> nosy: +eryksun
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue34115>
> _______________________________________
>
History
Date User Action Args
2018-07-24 19:43:22Yonatan Zungersetmessages: + msg322325
2018-07-22 09:12:01eryksunsetnosy: + eryksun
messages: + msg322139
2018-07-21 21:14:35Yonatan Zungersetmessages: + msg322123
2018-07-21 05:16:21terry.reedysetmessages: + msg322073
2018-07-21 01:43:17Yonatan Zungersetmessages: + msg322057
2018-07-21 01:09:37Yonatan Zungersetmessages: + msg322055
2018-07-20 23:40:36terry.reedysetassignee: docs@python
type: behavior -> enhancement
components: + Documentation
versions: + Python 3.8, - Python 3.7
nosy: + docs@python, terry.reedy

messages: + msg322039
stage: needs patch
2018-07-14 22:49:43Yonatan Zungercreate