classification
Title: Newline for print() is \n on Windows, and not \r\n as expected
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: M..Z., amaury.forgeotdarc, flox, georg.brandl, haypo, ishimoto, loewis, pitrou, python-dev, serhiy.storchaka
Priority: release blocker Keywords: patch

Created on 2011-10-06 20:59 by M..Z., last changed 2012-08-16 06:58 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
newline.py M..Z., 2011-10-06 20:59 Small trivial script
newline_3.1.txt M..Z., 2011-10-06 21:00 Py 3.1.4 output on Windows
newline_3.2.txt M..Z., 2011-10-06 21:00 Py 3.2.2 output on Windows
windows_stdout_newline.patch haypo, 2012-08-01 07:20 review
issue13119_test.patch ishimoto, 2012-08-02 05:20
issue13119_httpserver.patch ishimoto, 2012-08-04 23:41 review
issue13119_unbuffered.patch ishimoto, 2012-08-05 01:02 review
Messages (36)
msg145039 - (view) Author: M. Zilmer (M..Z.) Date: 2011-10-06 20:59
In 3.2.2 the newline for print() is \n on Windows, and not \r\n as expected.

In 3.1.4 the newline is \r\n.

OS is Win 7, and tried on both 32 and 64 bit.

Small example with output is attached.
msg145050 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-06 23:05
To people who open the file in their browser: text files are very similar, but newline_3.1.txt has CRLF line endings and newline_3.2.txt has LF line endings.

M.Z, how did you obtain them? did you start a subprocess?
msg145054 - (view) Author: M. Zilmer (M..Z.) Date: 2011-10-07 04:47
Hi Amaury,

The two text files were obtained through redirection in Windows, so I simply ran the newline.py file with:

    ...> C:\Python31\python.exe newline.py > newline_3.1.txt

    ...> C:\Python32\python.exe newline.py > newline_3.2.txt

Best regards,
Morten Zilmer
msg145064 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-07 08:52
I changed how newlines are handled on Windows to fix an issue with CGI: see the issue #10841.

changeset:   67431:0933c3753a71
user:        Victor Stinner <victor.stinner@haypocalc.com>
date:        Fri Jan 07 18:47:22 2011 +0000
files:       Misc/NEWS Modules/_io/fileio.c Modules/main.c Parser/tokenizer.c
description:
Issue #10841: set binary mode on files; the parser translates newlines

On Windows, set the binary mode on stdin, stdout, stderr and all
io.FileIO objects (to not translate newlines, \r\n <=> \n). The Python parser
translates newlines (\r\n => \n).
msg145066 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-07 09:01
print() uses PyFile_WriteString("\n", file) by default (if the end argument is not set) to write the newline. TextIOWrapper.write("\n") replaces "\n" by TextIOWrapper._writenl.

On Windows, stdin, stdout and stderr are creates using TextIOWrapper(..., newline=None). In this case, TextIOWrapper._writenl is os.linesep and so '\r\n'.

To sum up, print() writes '\n' into sys.stdout, but sys.stdout write b'\r\n' into the file descriptor 1 which is a binary file (ie. the underlying OS file doesn't translate newlines).

If the output is redirected (e.g. into a file), TextIOWrapper is created with line_buffering=False.

You may try to force line_buffering=True when the output is redirected.
msg145068 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-07 09:32
> If the output is redirected (e.g. into a file),
> TextIOWrapper is created with line_buffering=False.
How does this affect the \r\n translation?
msg145310 - (view) Author: M. Zilmer (M..Z.) Date: 2011-10-10 17:51
Just to make it clear: I have not observed any problems on the Windows terminal (cmd) with \n newline, but at least Notepad does not break lines correctly if only \n is used.
msg167067 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-01 01:45
I found 'more' command in Windows7 requires \r\n.

Python 2.7.3:

C:\>python -c "for i in range(5):print(i)"|more
0
1
2
3
4

Python 3.3(trunk):

c:\src\cpython\PCbuild>python -c "for i in range(5):print(i)"|more
?????
msg167092 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-08-01 07:19
> On Windows, stdin, stdout and stderr are creates using TextIOWrapper(..., newline=None).
> In this case, TextIOWrapper._writenl is os.linesep and so '\r\n'.

Oh, I was wrong: stdin is created with newline=None, but stdout and stderr are created with newline="\n" and so "\n" is not translated to "\r\n".

I checked in Python 2.7: print("abc") and sys.stdout.write("abc\n") writes b"abc\r\n" into the output file (when the output is redirected), but sys.stdout.write("abc\r\n") writes b"abc\r\r\n". Python 3.3 should do the same: \r\n is preferred on Windows (ex: notepad doesn't support UNIX line ending, \n).

Attached patch changes line ending for stdout and stderr on Windows: translate "\n" to "\r\n".

It would be nice to fix this before Python 3.3 final.
msg167193 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-02 05:20
Test for this issue. Tested on Windows7, Ubuntu linux 12.04.

I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.

But in Python2.7, "print >>sys.stderr, 1" also returns '1', 
so this might not be a problem.
msg167250 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-08-02 20:16
> I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.

I suppose that you mean "returns '1\n' instead of '1'". This is a
major change between Python 2 and Python 3. Use print(1, end=' ') if
you want the same behaviour. See:
http://docs.python.org/dev/whatsnew/3.0.html#print-is-a-function

You can also use the print() as a function in Python 2 using "from
__future__ import print_function":
http://docs.python.org/dev/whatsnew/2.6.html#pep-3105-print-as-a-function

I never liked "print expr," because it looks like a typo or an ugly
hack. It's easy to add a comma by mistake.
msg167257 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-08-02 21:11
> It would be nice to fix this before Python 3.3 final.

@Georg: So, what do you think?
msg167269 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-02 22:33
About the patch: why wouldn't you use newline = NULL in both cases?
msg167288 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-03 04:44
On Fri, Aug 3, 2012 at 5:16 AM, STINNER Victor <report@bugs.python.org> wrote:

>> I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.
>
> I suppose that you mean "returns '1\n' instead of '1'". 

No, sorry for my lame wording.

In the test I submitted, printing to stdout with 
    
    "print(1, file=sys.stdout);print(2, file=sys.stdout)"

outputs

    "1\r\n2\r\n"

but printing to stderr with 

    "print(1, file=sys.stderr);print(2, file=sys.stderr)" 

outputs

    "1\r\n2"   <- no '\r\n' at the end


I wondered why, but this is not specific to Python 3. 
With Python 2.7 

    print >>sys.stderr, 1

doesn't output '\r\n' at the end also. So I think this may 
not be a bug.
msg167379 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-03 23:31
New changeset c55dbb84f3b4 by Victor Stinner in branch 'default':
Close #13119: use "\r\n" newline for sys.stdout/err on Windows
http://hg.python.org/cpython/rev/c55dbb84f3b4
msg167382 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-03 23:42
New changeset 09408b990ca5 by Victor Stinner in branch '3.2':
Close #13119: use "\r\n" newline for sys.stdout/err on Windows
http://hg.python.org/cpython/rev/09408b990ca5
msg167449 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-04 22:11
Windows buildbots now show failures in the test suite.
msg167452 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-04 22:29
New changeset 4efad7fba42a by Antoine Pitrou in branch '3.2':
Fix test_sys under Windows (issue #13119)
http://hg.python.org/cpython/rev/4efad7fba42a

New changeset e4a87f0253e9 by Antoine Pitrou in branch 'default':
Merge universal newlines-related fixes (issue #13119)
http://hg.python.org/cpython/rev/e4a87f0253e9
msg167454 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-04 22:35
New changeset f8e435d6a801 by Antoine Pitrou in branch 'default':
Fix test_venv to work with universal newlines (issue #13119)
http://hg.python.org/cpython/rev/f8e435d6a801
msg167457 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-04 22:46
test_httpservers still fails, it's the CGI tests...
msg167460 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-04 23:41
Fix for test_httpservers
msg167465 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-05 00:40
Sorry, please ignore the patch 'issue13119_httpserver.patch' I posted above. 

Behavior of "-u" commandline option in Python3.3 is differ than in Python 2. 

We should not convert newline characters if "-u" specified? I'll investigate more.
msg167468 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-05 01:02
We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.

Added a patch to fix error in test_httpservers.
msg167479 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-05 10:26
> We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.

Why that? What do universal newlines have to do with buffering?
msg167481 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-05 10:44
I wonder why this is a release blocker. It's a bug in Python 3.2, so why should it block the release of 3.3 (it's not a regression).

If no complete solution is coming up, I recommend to revert all changes on this issue, and reconsider after the 3.3 release.
msg167482 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-05 10:49
> I wonder why this is a release blocker. It's a bug in Python 3.2, so
> why should it block the release of 3.3 (it's not a regression).

It's a blocker because the fix broke a couple of tests.
And it's also a regression from 3.1 and 2.7.
msg167485 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-05 11:03
> It's a blocker because the fix broke a couple of tests.

That cannot possibly be the explanation why haypo declared
it a blocker on 2012-08-01; the fix was only applied on
2012-08-04.

But I agree that it should block the release now; hence I
propose to roll back the entire set of changes and revert
to the 3.2 state.

> And it's also a regression from 3.1 and 2.7.

Since 3.2 was released with this behavior, this bug cannot manage
to block 3.3.
msg167486 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-05 11:16
Antoine Pitrou added the comment:
>
>> We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.
>
> Why that? What do universal newlines have to do with buffering?

Man page of Python says 

  -u Force  stdin,  stdout  and  stderr to be totally unbuffered.  On
     systems where it matters, also put stdin, stdout and  stderr  in
     binary  mode.   

test_httpservers depends on this behavior, but was implemented as documented in Python 3.
msg167488 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-05 11:21
> Man page of Python says 
> 
>   -u Force  stdin,  stdout  and  stderr to be totally unbuffered.  On
>      systems where it matters, also put stdin, stdout and  stderr  in
>      binary  mode.   

I don't know which version it is, but current 3.3 says:

“Force the binary I/O layers of stdin, stdout and  stderr  to  be
unbuffered.  The text I/O layer will still be line-buffered.”

> test_httpservers depends on this behavior, but was implemented as
> documented in Python 3.

I would argue that test_httpservers is wrong, since it uses print() in a
CGI script where sys.stdout.buffer.write() should really be used.
msg167499 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-05 12:56
New changeset bc4fdb758b8c by Antoine Pitrou in branch '3.2':
Fix CGI tests to take into account the platform's line ending (issue #13119)
http://hg.python.org/cpython/rev/bc4fdb758b8c

New changeset ee185c6b2880 by Antoine Pitrou in branch 'default':
Fix CGI tests to take into account the platform's line ending (issue #13119)
http://hg.python.org/cpython/rev/ee185c6b2880
msg167500 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-08-05 13:22
Buildbots are happy, the issue can be closed again. Thanks Antoine.
msg167502 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2012-08-05 13:28
> I don't know which version it is, but current 3.3 says:

Ah, sorry, I thought I was reading latest Man page.
msg168348 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-08-16 06:28
> +    "  newline is '' or '\n', no translation takes place. If newline is any\n"

Non-escaped "\n".
msg168349 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-16 06:33
Would be nice to be a bit more specific *where* that line comes from.
msg168350 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-08-16 06:54
> Would be nice to be a bit more specific *where* that line comes from.

Modules/_io/textio.c, changesets 243ad1a6f638 and 083776adcacc.
msg168351 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-08-16 06:58
Oops, I got the wrong issue, sorry.
History
Date User Action Args
2012-08-16 06:58:03serhiy.storchakasetstatus: open -> closed

messages: + msg168351
2012-08-16 06:54:33serhiy.storchakasetmessages: + msg168350
2012-08-16 06:33:05georg.brandlsetmessages: + msg168349
2012-08-16 06:28:05serhiy.storchakasetstatus: closed -> open
nosy: + serhiy.storchaka
messages: + msg168348

2012-08-05 13:28:05ishimotosetmessages: + msg167502
2012-08-05 13:22:34hayposetstatus: open -> closed

messages: + msg167500
2012-08-05 12:56:32python-devsetmessages: + msg167499
2012-08-05 11:21:18pitrousetmessages: + msg167488
2012-08-05 11:16:13ishimotosetmessages: + msg167486
2012-08-05 11:03:28loewissetmessages: + msg167485
2012-08-05 10:49:09pitrousetmessages: + msg167482
2012-08-05 10:44:32loewissetnosy: + loewis
messages: + msg167481
2012-08-05 10:26:05pitrousetmessages: + msg167479
2012-08-05 01:02:01ishimotosetfiles: + issue13119_unbuffered.patch

messages: + msg167468
2012-08-05 00:40:50ishimotosetmessages: + msg167465
2012-08-04 23:41:42ishimotosetfiles: + issue13119_httpserver.patch

messages: + msg167460
2012-08-04 22:46:32pitrousetmessages: + msg167457
2012-08-04 22:35:52python-devsetmessages: + msg167454
2012-08-04 22:29:44python-devsetmessages: + msg167452
2012-08-04 22:11:14pitrousetstatus: closed -> open

messages: + msg167449
versions: + Python 3.3
2012-08-03 23:42:19python-devsetmessages: + msg167382
2012-08-03 23:31:57python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg167379

resolution: fixed
stage: resolved
2012-08-03 04:44:28ishimotosetmessages: + msg167288
2012-08-02 22:33:39pitrousetmessages: + msg167269
2012-08-02 21:11:04hayposetmessages: + msg167257
2012-08-02 20:16:45hayposetmessages: + msg167250
2012-08-02 05:20:09ishimotosetfiles: + issue13119_test.patch

messages: + msg167193
2012-08-01 07:20:01hayposetfiles: + windows_stdout_newline.patch
priority: normal -> release blocker

nosy: + georg.brandl
messages: + msg167092

keywords: + patch
2012-08-01 01:45:03ishimotosetnosy: + ishimoto
messages: + msg167067
2011-10-10 17:51:49M..Z.setmessages: + msg145310
2011-10-09 21:13:38floxsetnosy: + flox
2011-10-07 09:32:44amaury.forgeotdarcsetmessages: + msg145068
2011-10-07 09:01:18hayposetnosy: + pitrou
messages: + msg145066
2011-10-07 08:52:32hayposetmessages: + msg145064
2011-10-07 04:47:56M..Z.setmessages: + msg145054
2011-10-06 23:05:47amaury.forgeotdarcsetnosy: + amaury.forgeotdarc, haypo
messages: + msg145050
2011-10-06 21:00:38M..Z.setfiles: + newline_3.2.txt
2011-10-06 21:00:22M..Z.setfiles: + newline_3.1.txt
2011-10-06 20:59:47M..Z.create