classification
Title: In IDLE, sys.stdout and sys.stderr can write any pickleable object
Type: crash Stage:
Components: IDLE Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Ramchandra Apte, loewis, orsenthil, python-dev, roger.serwy, serhiy.storchaka, terry.reedy
Priority: high Keywords: patch

Created on 2011-12-05 14:07 by Ramchandra Apte, last changed 2012-07-10 18:38 by loewis. This issue is now closed.

Files
File name Uploaded Description Edit
stdout_fix1.diff roger.serwy, 2012-03-11 21:53 review
issue13532_27.patch roger.serwy, 2012-07-09 19:26 review
stdin_fix.patch roger.serwy, 2012-07-10 14:38 review
Messages (34)
msg148873 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2011-12-05 14:07
In IDLE, sys.stdout.write and sys.stderr can write any pickleable object such as 100 when they should only allow strings.
IDLE seems to be pickling the object.
>>> import sys
>>> sys.stdout.write(100)
100
>>> sys.stdout.write(sys)
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    sys.stdout.write(sys)
_pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup builtins.module failed
The error above is more detailed in IDLE 2.7.
While in Python on the command-line:
>>> import sys
>>> sys.stdout.write(100)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be str, not int
>>> 
The error above in Python 2.7:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: expected a character buffer object
msg148889 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-12-05 21:27
Why do you think this is a bug? Is this behavior causing problems?
msg148907 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2011-12-06 04:59
It is different behaviour than usual.
I agree it is of low importance.
msg149361 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-12 22:34
We should like the IDLE shell to give the same results as the standard shell. See #7163

On 3.2.2 on Win7, the problem is worse: sys.stdout.write(100) crashes IDLE -- as in it just fades away after a short (1/2 sec?) delay.
 sys.stdout.write(sys)
gives me the error reported.

The difference comes from this in the Command Window:
>>> sys.stdout
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp437'>

versus this in IDLE:
>>> import sys
>>> sys.stdout
<idlelib.rpc.RPCProxy object at 0x00000000031D0208>
msg149370 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2011-12-13 01:27
The crash can be prevented by using #13582.

The RPCProxy object would need to be subclassed so that it raises TypeError for the "write" method when it is not given a string.
msg149459 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-12-14 17:38
> We should like the IDLE shell to give the same results as the standard shell. 

I disagree that this should be an absolute principle. Two standard
shells may not give the same result due to running in different
environments, so forcing IDLE to give the same result as a specific
standard shell is not reasonable when taken absolutely.
msg149470 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-14 20:18
In English, "We should like" is far from an absolute statement. "IDLE must" would be such. In any case, I hope you agree that crashing is bad. It is also not good if IDLE, as a development environment, enables code that violates the doc specifications and that will not run in the standard interpreter in standard batch mode with no shell.

I also so not understand you reverting the title to the arguably incorrect non-parallel construction.
msg149475 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-12-14 20:59
> I also so not understand you reverting the title to the arguably incorrect non-parallel construction.

That is an (unfortunate) side effect of replying by email.
msg149490 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2011-12-15 03:05
One reason to fix this bug:

People may develop code that calls sys.std{out,err}.write with the number 200.
like sys.stdout.write(200)
In IDLE the code works well but in Python from the command-line it fails to run.
Creating a bug in the code.
msg150075 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2011-12-22 07:22
This bug is related to #1757057.
msg155285 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-03-10 03:01
What is the status of this bug?
It has been about two months since the last message.
msg155291 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-03-10 04:54
Many issues take a while, depending on volunteers time and choice of priority.
msg155295 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-03-10 07:01
Ramchandra: Most likely, the status is that nobody is working on this, and nobody is planning to work on this. So if you want to see it fixed, it may be that you have to submit a patch yourself.
msg155411 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-03-11 21:53
Attached is a preliminary patch to solve this issue. It relies on "rpc_marshal_exception.patch" from issue14200. 

Does anyone know a good way to make the exception render as: "must be str, not int" instead of "must be str, not <class 'int'>" ?
msg155440 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-03-12 04:18
Just a minor comment on the patch - It should be %r instead of %s.

-            raise TypeError('must be str, not %s' % type(s))
+            raise TypeError('must be str, not %r' % type(s))

If there is any test for this, it could be added. I find this approach 'OK'.
msg162527 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-06-08 13:40
@Roger Serwy
>> Does anyone know a good way to make the exception render as: "must be str, not int" instead of "must be str, not <class 'int'>" ?
Use s.__name__ rather than type(s) for the type name
e.g.
-raise TypeError('must be str, not %s' % type(s))
+raise TypeError('must be str, not %s' % s.__name__)
msg162645 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-06-11 21:13
@Ramchandra: "__name__" does not exist for many objects.

This issue with the sys.stdout.write encompasses a lot of other issues involving the shortcomings of the RPCProxy object. The following code prevents another prompt from appearing:

    >>> class A:
            pass

    >>> import sys
    >>> sys.stdout.write(A)

Even though "A" is pickleable, IDLE gets stuck trying to write to stdout. In run.py, Executive.runcode gets stuck on "exec(code, self.locals)", and is not interruptable. Restarting the shell makes the IDLE shell responsive again to commands.
msg164286 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-06-28 17:57
> Does anyone know a good way to make the exception render as: "must be str, not int" instead of "must be str, not <class 'int'>" ?

raise TypeError('must be str, not %s' % type(s).__name__)
msg164287 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-06-28 18:02
> The following code prevents another prompt from appearing:

And this a more serious issue concerns 3.3.
msg165062 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-07-09 05:09
The patch (to my production Idle, with name fix) prevents crashing and raises an error instead when an object *can* be pickled, so I will apply before 3.3 if no problems appear and we cannot do better.

But written objects are still pickled, so sys.stdout.write(sys) still raises the PicklingError instead of (as in CP interpreter)
  TypeError: must be str, not module

It seems to me that the type check should be done in the subprocess before the object (which should be a string) is pickled. (I also wonder if it is really necessary to pickle a string or the encoded bytes to send it back. The pickle is just a stream of bytes.)
msg165107 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-07-09 18:53
New changeset 422242dbce30 by Martin v. Löwis in branch '3.2':
Issue #13532: Check that arguments to sys.stdout.write are strings.
http://hg.python.org/cpython/rev/422242dbce30
msg165108 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-07-09 19:02
New changeset 58189e37331c by Martin v. Löwis in branch '2.7':
- Issue #13532: Check that arguments to sys.stdout.write are strings.
http://hg.python.org/cpython/rev/58189e37331c
msg165110 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-07-09 19:05
All problems in computer science can be solved by another indirection...

I've added another wrapper around the RPC proxy to cause the type error. (A variant of) Roger's patch is still included to support the -n case.
msg165112 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-07-09 19:26
I just pulled Martin's patches and they fix the issue. Indirection to the rescue!

There is one very slight problem though with the error message raised on 2.7 since it is different than 3.3. Attached is a patch to fix it.
msg165123 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-07-09 20:28
Ah, this indicates that we should support any buffer object...
msg165124 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-07-09 20:45
Thanks Martin. I was thinking of adding special-casing inside the rpcproxy class, but the wrapping is pretty clean and clear. Works great in 3.3 Win7-64.

The 2.7 message 'expected a character buffer object' is technically correct*, but opaque, especially to beginners. I actually prefer the 3.x style message, except that 'str' should be expanded to 'string'. (We especially should not say 'character buffer object' until we test for exactly that.)

*My impression is that 2.7 unicode is not a character buffer object, but gets auto converted to str/bytes, which is.

On the technically correct front, bytearray in 2.7 is a character buffer object that .write() can write, but it is not a subclass of basestring. So the widened test in the followup patch
http://hg.python.org/cpython/rev/2993f566c82e
should be widened further.

-        if not isinstance(s, basestring):
-             raise TypeError('must be str, not ' + type(s).__name__)

+        if not isinstance(s, (basestring, bytearray)):
+             raise TypeError('must be string, not ' + type(s).__name__)

Are there any other 'character buffer' classes that should be included? What *is* the test that 2.7 .write() uses to determine what is such? Can it even be written in Python (rather than C)?
msg165125 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-07-09 20:48
I'm losing interest in this issue. It already got more attention than it deserves. Terry, feel free to make whatever change you consider desirable.
msg165129 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-07-09 21:58
New changeset 4f891f44ec15 by Terry Jan Reedy in branch '2.7':
Issue 13532: Allow bytearrays to be written also.
http://hg.python.org/cpython/rev/4f891f44ec15
msg165130 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-07-09 22:01
I tested and pushed the change to allow bytestrings. If anyone is printing other character buffer objects in 2.7 with Idle, they can explicitly convert to str/bytes, patch run.py, and/or post here or on a new issue for further expansion. I am otherwise done with this too.
msg165168 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-07-10 06:40
> +        sys.stdin = self.console = _RPCFile(self.get_remote_proxy("stdin"))

write to sys.stdin?
msg165189 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-07-10 14:10
sys.stdin has a write method, but it should raise "io.UnsupportedOperation: not writable" when passed a string. It looks like IDLE has allowed writes to stdin even before Martin's patch. I'll open a separate issue for this case.
msg165191 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-07-10 14:38
Running "input" from IDLE now raises an error. Attached is a fix to _RPCFile to allow readline (and isatty) to function properly.
msg165194 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-07-10 15:36
>>> sys.stdin.readable()
False
>>> sys.stdout.writable()
False

I think the issue is far from a solution.
msg165197 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-07-10 18:38
Folks: one issue at a time - pretty pretty please.

This issue was "In IDLE, sys.stdout and sys.stderr can write any pickleable object". Will anybody dispute that this issue is fixed?

Roger: please create a new issue for input being broken now. Your patch "shouldn't" be necessary, as __getattr__ should deal with this.

Serhiy: please create a new issue for readable and writable returning an incorrect result.
History
Date User Action Args
2012-07-10 18:38:56loewissetstatus: open -> closed
resolution: fixed
messages: + msg165197
2012-07-10 15:36:50serhiy.storchakasetmessages: + msg165194
2012-07-10 14:38:01roger.serwysetstatus: closed -> open
files: + stdin_fix.patch
messages: + msg165191

resolution: fixed -> (no value)
priority: normal -> high
2012-07-10 14:10:01roger.serwysetmessages: + msg165189
2012-07-10 06:40:13serhiy.storchakasetmessages: + msg165168
2012-07-09 22:01:01terry.reedysetmessages: + msg165130
2012-07-09 21:58:16python-devsetmessages: + msg165129
2012-07-09 20:48:50loewissetmessages: + msg165125
2012-07-09 20:45:13terry.reedysetmessages: + msg165124
2012-07-09 20:28:35loewissetmessages: + msg165123
2012-07-09 19:26:58roger.serwysetfiles: + issue13532_27.patch

messages: + msg165112
2012-07-09 19:05:06loewissetstatus: open -> closed
resolution: fixed
messages: + msg165110
2012-07-09 19:02:00python-devsetmessages: + msg165108
2012-07-09 18:53:18python-devsetnosy: + python-dev
messages: + msg165107
2012-07-09 05:09:45terry.reedysetmessages: + msg165062
title: In IDLE, sys.stdout.write and sys.stderr can write any pickleable object -> In IDLE, sys.stdout and sys.stderr can write any pickleable object
2012-06-28 18:02:01serhiy.storchakasetmessages: + msg164287
versions: + Python 3.3
2012-06-28 17:57:24serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg164286
2012-06-11 21:13:21roger.serwysetmessages: + msg162645
2012-06-08 13:40:26Ramchandra Aptesetmessages: + msg162527
2012-03-12 04:18:48orsenthilsetnosy: + orsenthil
messages: + msg155440
2012-03-11 21:53:02roger.serwysetfiles: + stdout_fix1.diff
keywords: + patch
messages: + msg155411
2012-03-10 07:01:08loewissetmessages: + msg155295
2012-03-10 04:54:17terry.reedysetmessages: + msg155291
2012-03-10 03:01:16Ramchandra Aptesetmessages: + msg155285
2011-12-22 07:22:17roger.serwysetmessages: + msg150075
2011-12-15 03:05:08Ramchandra Aptesetmessages: + msg149490
2011-12-14 20:59:09loewissetmessages: + msg149475
2011-12-14 20:18:25terry.reedysetmessages: + msg149470
2011-12-14 17:38:51loewissetmessages: + msg149459
title: In IDLE, sys.stdout and sys.stderr can write any pickleable object -> In IDLE, sys.stdout.write and sys.stderr can write any pickleable object
2011-12-14 02:48:48Ramchandra Aptesettitle: In IDLE, sys.stdout.write and sys.stderr can write any pickleable object -> In IDLE, sys.stdout and sys.stderr can write any pickleable object
2011-12-13 01:27:09roger.serwysetnosy: + roger.serwy
messages: + msg149370
2011-12-12 22:34:17terry.reedysettype: behavior -> crash

messages: + msg149361
nosy: + terry.reedy
2011-12-06 04:59:54Ramchandra Aptesetmessages: + msg148907
2011-12-05 21:27:49loewissetnosy: + loewis
messages: + msg148889
2011-12-05 14:07:42Ramchandra Aptecreate