classification
Title: input() has trailing carriage return on windows
Type: behavior Stage: test needed
Components: Interpreter Core, Windows Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Phoenix Fire, SilentGhost, brian.curtin, duncanb, ezio.melotti, georg.brandl, haypo, pitrou, r.david.murray, v+python
Priority: critical Keywords: 3.2regression, patch

Created on 2011-02-21 15:48 by duncanb, last changed 2011-05-25 15:16 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
issue11272.patch haypo, 2011-02-23 01:11
stdintests.py duncanb, 2011-02-23 09:20
unnamed Phoenix Fire, 2011-05-25 10:32
Messages (17)
msg128964 - (view) Author: Duncan Booth (duncanb) Date: 2011-02-21 15:48
In Python 3.2, the builtin function `input()` returns a string with a trailing '\r' on windows:

    C:\Python32>python
    Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print(repr(input()))
    test
    'test\r'
    >>>

This breaks code that expects the string to be stripped, e.g. 'pydoc.py -b' doesn't recognise its commands:

    C:\Python32>python lib\pydoc.py -b
    Server ready at http://localhost:4680/
    Server commands: [b]rowser, [q]uit
    server> q
    Server commands: [b]rowser, [q]uit
    server> b
    Server commands: [b]rowser, [q]uit
    server>
msg128967 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-02-21 16:10
Confirmed on Python 3.2 (winxp). The problem doesn't seem to exist on 3.1.3.
msg128971 - (view) Author: SilentGhost (SilentGhost) Date: 2011-02-21 16:49
On WinXp with Python 3.2a4+ or 3.1.3 I cannot reproduce this issue.
msg128972 - (view) Author: SilentGhost (SilentGhost) Date: 2011-02-21 17:10
With py3.2 final, I can reproduce this bug with command line (as demonstrated by the OP) but not with the IDLE (for 3.2a4+ I have only command line, which I compiled myself).
msg128973 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-02-21 17:13
#10841 may be related.
msg128981 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-21 19:20
> Confirmed on Python 3.2 (winxp).
> The problem doesn't seem to exist on 3.1.3.

Can you try Python 3.1 with -u command line flag?

I changed Python 3.2 to always open all files in binary module, not only if -u flag is used. I had also to fix the parser to support \r\n newlines: it looks like I missed something in the parser.
msg128982 - (view) Author: SilentGhost (SilentGhost) Date: 2011-02-21 19:27
> Can you try Python 3.1 with -u command line flag?
Yes, I can reproduce it with 3.1.3 with -u flag
msg128988 - (view) Author: Duncan Booth (duncanb) Date: 2011-02-21 20:05
Yes, it does indeed look like stdin has been opened in binary mode. Just iterating over it also gives the spurious carriage returns:


    C:\Python32>python
    Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> for line in sys.stdin:
    ...     print(repr(line))
    ...
    hello
    'hello\r\n'
    ^Z
    >>>
msg129144 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-23 00:40
Here is a patch to fix input() on Windows: strip also \r.
msg129145 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-02-23 01:02
Is it possible to add some tests for input()?

Also the patch uses tabs instead of spaces.
msg129147 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-23 01:11
C:\Python32>python
    Python 3.2 ... on win32
    >>> import sys
    >>> for line in sys.stdin:
    ...     print(repr(line))
    ...
    hello
    'hello\r\n'
    ^Z

Oh yes, I confirm that there is a second bug: sys.stdin doesn't translate \r\n to \n, whereas sys.stdin is a text file.

Attached patch fixes both issues.

> Is it possible to add some tests for input()?

input() has already tests, but the bug was not detected because the test uses a mockup, not a subprocess. Moreover, I don't think that it is possible to test input() for the TTY case. It is not easy to test a TTY, especially on Windows.

If anyone knows how to reproduce the two bugs with a short Python script, I can try to convert it into a test.

> Also the patch uses tabs instead of spaces.

Yeah, I hate producing patches on Windows. Fixed in the new patch (prepared on Linux).
msg129160 - (view) Author: Duncan Booth (duncanb) Date: 2011-02-23 09:20
> If anyone knows how to reproduce the two bugs with a short Python
> script, I can try to convert it into a test.

If you don't mind kicking off some sub-processes then here's a script that shows the bugs.

I couldn't figure out how to do a script that would work on Python 3.1 but fail on Python 3.2, because I think to show the problem you have to use the shell to pipe data and Popen on Python 3.1 quotes the pipe character.
msg129180 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-23 12:11
Fixed in 3.3 (r88530) and 3.2 (r88531). Others versions are not affected.

Thanks Duncan Booth, I added tests based on your stdintests.py script. I used directly stdin argument of Popen() instead of using cmd.exe to create the pipe (which is not portable).
msg136835 - (view) Author: Giuseppe Laurenza (Phoenix Fire) Date: 2011-05-25 10:02
On my pc with both architecture (x86 and x64) the bug is still presentes
msg136836 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-25 10:16
> On my pc with both architecture (x86 and x64) the bug
> is still presentes

Which version of Python are you using? Python 3.1, Python 3.2.1 and Python 3.3 doesn't have the bug, Python 3.2.0 has the bug. Python 3.2.1 doesn't have the bug, but it's not released yet: it will be released in a few weeks. Python 3.3 is a development version.
msg136837 - (view) Author: Giuseppe Laurenza (Phoenix Fire) Date: 2011-05-25 10:32
I'm using the 3.2,
the 3.0 has the bug?

2011/5/25 STINNER Victor <report@bugs.python.org>

>
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
>
> > On my pc with both architecture (x86 and x64) the bug
> > is still presentes
>
> Which version of Python are you using? Python 3.1, Python 3.2.1 and Python
> 3.3 doesn't have the bug, Python 3.2.0 has the bug. Python 3.2.1 doesn't
> have the bug, but it's not released yet: it will be released in a few weeks.
> Python 3.3 is a development version.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11272>
> _______________________________________
>
msg136854 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-05-25 15:16
You are using the only version that has the bug (3.2.0, also called 3.2).  The fixed version will be released shortly (3.2.1).
History
Date User Action Args
2011-07-03 11:05:50georg.brandllinkissue12477 superseder
2011-06-28 22:42:09amaury.forgeotdarclinkissue12435 superseder
2011-05-25 15:16:35r.david.murraysetnosy: + r.david.murray
messages: + msg136854
2011-05-25 10:32:15Phoenix Firesetfiles: + unnamed

messages: + msg136837
2011-05-25 10:16:09hayposetmessages: + msg136836
2011-05-25 10:02:02Phoenix Firesetnosy: + Phoenix Fire
messages: + msg136835
2011-03-07 17:30:43r.david.murraylinkissue11434 superseder
2011-02-23 12:11:58hayposetstatus: open -> closed

messages: + msg129180
resolution: fixed
nosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
2011-02-23 09:20:36duncanbsetfiles: + stdintests.py
nosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
messages: + msg129160
2011-02-23 01:11:32hayposetfiles: - input_rn.patch
nosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
2011-02-23 01:11:05hayposetfiles: + issue11272.patch
nosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
messages: + msg129147
2011-02-23 01:02:33ezio.melottisetnosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
messages: + msg129145
2011-02-23 00:40:53hayposetfiles: + input_rn.patch

messages: + msg129144
keywords: + patch
nosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
2011-02-21 20:05:24duncanbsetnosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
messages: + msg128988
2011-02-21 19:27:29SilentGhostsetnosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
messages: + msg128982
2011-02-21 19:20:24hayposetnosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
messages: + msg128981
2011-02-21 18:07:19georg.brandlsetkeywords: + 3.2regression
nosy: duncanb, georg.brandl, pitrou, haypo, ezio.melotti, v+python, brian.curtin, SilentGhost
2011-02-21 17:30:15SilentGhostsetnosy: + haypo
2011-02-21 17:14:45ezio.melottisetnosy: + pitrou, v+python
2011-02-21 17:13:12brian.curtinsetnosy: duncanb, georg.brandl, ezio.melotti, brian.curtin, SilentGhost
messages: + msg128973
2011-02-21 17:10:57SilentGhostsetnosy: duncanb, georg.brandl, ezio.melotti, brian.curtin, SilentGhost
messages: + msg128972
2011-02-21 16:49:39SilentGhostsetnosy: + SilentGhost
messages: + msg128971
2011-02-21 16:10:44brian.curtinsetnosy: + brian.curtin
2011-02-21 16:10:16ezio.melottisetpriority: normal -> critical

messages: + msg128967
stage: test needed
2011-02-21 15:54:25ezio.melottisetnosy: + georg.brandl, ezio.melotti
2011-02-21 15:48:24duncanbcreate