classification
Title: Can't assign a different value for sys.stdin in IDLE
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Guilherme.Simões, Todd.Rovito, benjamin.peterson, georg.brandl, larry, python-dev, roger.serwy, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2013-04-25 06:56 by Guilherme.Simões, last changed 2013-06-12 02:48 by roger.serwy. This issue is now closed.

Files
File name Uploaded Description Edit
hold_stdin.patch roger.serwy, 2013-04-26 14:25 review
hold_stdin_rev1.patch roger.serwy, 2013-04-28 07:36 review
Terry17838.diff terry.reedy, 2013-04-28 17:47 review
Terry17838_rev1.patch roger.serwy, 2013-05-03 04:50 review
Messages (25)
msg187759 - (view) Author: Guilherme Simões (Guilherme.Simões) * Date: 2013-04-25 06:56
Something like:

sys.stdin = open('file')

works in Python but doesn't in the IDLE shell. After trying to do that, a dialog is open asking if the user wants to kill the program. This happens because the method "close" of the class "PseudoInputFile" (which was created in #17585) is called.
msg187776 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-25 11:29
Once again, what system and version? The Idle user process is different on *nix and Windows -- python.exe versus pythonw.exe. In normal interactive mode, the interpreter continues to take statements from the console, which seems to be sys.__stdin__, after redirection. Sys.stdin is only used for input() statements and direct sys.stdin reads.

Python 3.3.1 (v3.3.1:d9893d13c628, Apr  6 2013, ...
>>> import sys
>>> sys.__stdin__
<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp437'>
>>> sys.stdin = open('c:/python/mypy/tem.py')
>>> sys.stdin
<_io.TextIOWrapper name='c:/python/mypy/tem.py' mode='r' encoding='cp1252'>
>>> input()
'a, b = 1, 2'
>>> input()
'print("{a} apple and {b} bananas".format(**locals()))'
# These are the 'current' first two lines of the file.
>>> sys.__stdin__
<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp437'>

With Idle, at least on Windows, sys.__stdin__ is None. If sys.stdin is the only reference to PseudoInputFile, rebinding leads to closure. With no connection to the Shell window, it would be impossible to send statements.

>>> import sys
>>> sys.stdin
<idlelib.PyShell.PseudoInputFile object at 0x0000000002CD8780>
>>> sys.__stdin__
>>> 

However, with 3.3.1 on Windows, I do not reproduce the problem.
>>> sys.stdin = open('c:/python/mypy/tem.py')
>>> sys.stdin
<_io.TextIOWrapper name='c:/python/mypy/tem.py' mode='r' encoding='cp1252'>
>>> input()
'a, b = 1, 2'
>>> input()
'print("{a} apple and {b} bananas".format(**locals()))'

So pythonw.exe seems to keep a reference other than either .stdin or .__stdin__.
msg187784 - (view) Author: Guilherme Simões (Guilherme.Simões) * Date: 2013-04-25 13:37
I forgot to say I tested this in MacOS in the development version. It shouldn't happen in earlier versions because the method "close" of the class "PseudoInputFile" is not there. A quick fix to this bug would be to just remove this method, but then #17585 would have to be reopened.
msg187785 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-04-25 14:27
Are you running with or without a subprocess?
msg187791 - (view) Author: Guilherme Simões (Guilherme.Simões) * Date: 2013-04-25 16:23
Roger, I was running with a subprocess but I tried without one and now it works.
msg187815 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-04-25 20:24
I confirm this bug.

Terry, 3.3.1 doesn't contains this bug. This is a regression introduced by issue17585.

A right solution is not easy. We should

1) Remove PseudoInputFile.close.
2) Do not print an exception in Executive.runcode, but transfer it via rpc back to PyShell.
3) In PyShell.runcode receive an exception and process it same way as in non-subprocess mode (note that in non-subrocess mode a messagebox asks "Exit? Do you want to exit altogether?", but in subprocess mode it now asks "Kill? The program is still running! Do you want to kill it?").

The hard part is transferring an exception.
msg187857 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-04-26 14:25
If we remove .close() then we'll need to then have an alternative way to allow "exit()" and "quit()" to actually close IDLE. The RPC proxy already transfers exceptions between the subprocess and the front-end, so we could catch the SystemExit exception and handle it by calling close(). That's perfectably doable, and likely the right way to do it. If anyone wants to write that patch, I'll gladly offer guidance if necessary.

Here's a dirty hack that also solves the issue. Garbage collection on the PseudoInputFile for sys.stdin calls .close(). As long as we hold an extra reference to it, it won't close. The attached patch holds an extra reference to sys.stdin in Lib/idlelib/run.py.
msg187864 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-04-26 15:21
This issue is not important enough. We can defer fixing until a better patch will be proposed.

In any case please add an explaining comment to your patch when committing it. Otherwise some future code will depend on self.stdin.
msg187882 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-26 22:22
I think PseudoXxxFile should have close methods, but that the methods should not be called when they should not be. Rebinding sys.stdin should not close command line input.

My perhaps naive thought is that initializing sys.__stdin__ to the original sys.stdin, as in the standard interpreter and as specified in the docs, is the proper place for the extra reference to keep PseudoInputFile alive. (I presume we are talking about the object in the user process.) While the sys module doc says that any of sys.std** and sys.__std**__ can be None on Windows pythonw gui apps, that does not make None a good idea ;-).
msg187967 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-04-28 07:36
I agree with Serhiy that the patch should be updated to better explain why the extra reference to stdin was being held. The attached patch provides that update in case anyone considers applying it in the future.

Terry, are you suggesting that the code should read like "sys.__stdin__ = sys.stdin" within MyHandler in Lib/idlelib/run.py ?
msg187973 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-04-28 10:45
Last patch LGTM.

With keeping a reference in sys.__stdin__ we will encounter same issue.

    sys.stdin = None
    sys.__stdin__ = None  # close!
msg188009 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-28 17:47
Roger: yes. This solves immediate problem and makes Idle more like console interpreter. Same should be done for all three.

Serhiy: the sole and documented purpose of sys.__stdxyz__ is to serve as backup bindings for the three i/o streams, so rebinding them is senseless. I would be willing to say 'you get what you deserve if you do that'. But since console tolerates double rebinding, we can make Idle do so too. Again, same should be done for all three streams.

See attached patch. It seems to work, but someone please recheck for typos. Are we sure that binding streams to handler object cannot cause problems, such as during shutdown with atexit handlers?
msg188178 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-04-30 19:24
Due to the coming of regression fix releases the fix for this issue urgently needed. It fixes a regression in issue17585 which fixes a regression in last bugfix releases.
msg188290 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-05-03 04:50
Keeping the sys.stdin reference alive and then reassigning sys.stdin prevents exit() and quit() from actually closing IDLE since site.py's code closes sys.stdin which then does not trigger PyShell's close() method. I updated Terry's patch to explicitly call the close method when SystemExit gets raised.

IDLE's subprocess clears atexit handlers before shutdown so they never get called. See exit() from Lib/idlelib/run.py.

The debugger can not handle stepping through a "raise SystemExit" from the shell, even long before any sys.std* patches got applied. That's a separate issue that should be opened. I mention that so that if anyone manually tests with the debugger that this behavior is "expected".
msg188292 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-05-03 05:37
Is this applicable to 3.2?
msg188301 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-05-03 13:53
3.2 still has the problem fixed in issue17585 for 3.3 and 3.4. This only applies if issue17585 patch gets applied to 3.2.
msg188972 - (view) Author: Roundup Robot (python-dev) Date: 2013-05-12 03:24
New changeset 4b3d18117813 by Benjamin Peterson in branch '2.7':
prevent IDLE from trying to close when sys.stdin is reassigned (#17838)
http://hg.python.org/cpython/rev/4b3d18117813
msg188973 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-05-12 03:25
I applied the least controversial patch, which fixes the regression, so 2.7.5 can be released. Feel free to tweak as needed.
msg188993 - (view) Author: Roundup Robot (python-dev) Date: 2013-05-12 09:24
New changeset 5f62c848f713 by Benjamin Peterson in branch '3.3':
prevent IDLE from trying to close when sys.stdin is reassigned (#17838)
http://hg.python.org/cpython/rev/5f62c848f713

New changeset bc322854c336 by Georg Brandl in branch 'default':
Issue #17838: merge with 3.3
http://hg.python.org/cpython/rev/bc322854c336
msg188997 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-12 10:28
3.2?
msg189007 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-05-12 10:41
Not necessary as per msg188301.
msg189040 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-12 14:52
I mean applying both issue17585 and issue17838 patches.
msg189091 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-05-13 03:50
Georg, I need to clarify what I meant in msg188301. Issue9290 was applied to 3.2 which introduced the bug addressed in issue17585, whose fix then introduced issue17838.

IDLE on the 3.2 branch doesn't handle exit() or quit() correctly. It needs both #17585 and #17838 to be applied, as Serhiy said.
msg189102 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-05-13 06:15
Sorry, this was not clear to me.  3.2 users will have to live with the bug.
msg191003 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-06-12 02:48
I'm closing this issue as the original problem reported has been resolved.
History
Date User Action Args
2013-06-12 02:48:28roger.serwysetstatus: open -> closed

messages: + msg191003
stage: needs patch -> resolved
2013-05-13 06:15:05georg.brandlsetmessages: + msg189102
2013-05-13 03:50:42roger.serwysetmessages: + msg189091
2013-05-12 14:52:01serhiy.storchakasetmessages: + msg189040
2013-05-12 10:41:32georg.brandlsetmessages: + msg189007
2013-05-12 10:28:42serhiy.storchakasetmessages: + msg188997
2013-05-12 09:24:27python-devsetmessages: + msg188993
2013-05-12 03:25:58benjamin.petersonsetpriority: release blocker -> normal

messages: + msg188973
2013-05-12 03:24:51python-devsetnosy: + python-dev
messages: + msg188972
2013-05-03 13:53:52roger.serwysetmessages: + msg188301
2013-05-03 05:37:00georg.brandlsetmessages: + msg188292
2013-05-03 04:50:06roger.serwysetfiles: + Terry17838_rev1.patch

messages: + msg188290
2013-04-30 19:24:42serhiy.storchakasetpriority: normal -> release blocker
versions: + Python 3.2
nosy: + larry, benjamin.peterson, georg.brandl

messages: + msg188178
2013-04-28 17:47:32terry.reedysetfiles: + Terry17838.diff

messages: + msg188009
2013-04-28 10:45:33serhiy.storchakasetmessages: + msg187973
2013-04-28 07:36:27roger.serwysetfiles: + hold_stdin_rev1.patch

messages: + msg187967
2013-04-26 22:22:06terry.reedysetmessages: + msg187882
2013-04-26 15:21:50serhiy.storchakasetmessages: + msg187864
2013-04-26 14:25:03roger.serwysetfiles: + hold_stdin.patch
keywords: + patch
messages: + msg187857
2013-04-25 20:24:30serhiy.storchakasetversions: + Python 2.7, Python 3.3
nosy: + serhiy.storchaka

messages: + msg187815

stage: needs patch
2013-04-25 16:23:05Guilherme.Simõessetmessages: + msg187791
2013-04-25 14:27:21roger.serwysetmessages: + msg187785
2013-04-25 13:37:42Guilherme.Simõessetmessages: + msg187784
2013-04-25 11:29:59terry.reedysetmessages: + msg187776
2013-04-25 06:59:34Guilherme.Simõessetnosy: + terry.reedy, roger.serwy, Todd.Rovito
2013-04-25 06:56:24Guilherme.Simõescreate