Newline for print() is \n on Windows, and not \r\n as expected #57328

MZ · 2011-10-06T20:59:47Z

BPO	13119
Nosy	@loewis, @birkenfeld, @atsuoishimoto, @amauryfa, @pitrou, @vstinner, @florentx, @serhiy-storchaka
Files	newline.py: Small trivial script newline_3.1.txt: Py 3.1.4 output on Windows newline_3.2.txt: Py 3.2.2 output on Windows windows_stdout_newline.patch issue13119_test.patch issue13119_httpserver.patch issue13119_unbuffered.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-08-16.06:58:03.911>
created_at = <Date 2011-10-06.20:59:47.493>
labels = ['type-bug', 'OS-windows', 'release-blocker']
title = 'Newline for print() is \\n on Windows, and not \\r\\n as expected'
updated_at = <Date 2012-08-16.06:58:03.909>
user = 'https://bugs.python.org/MZ'

bugs.python.org fields:

activity = <Date 2012-08-16.06:58:03.909>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2012-08-16.06:58:03.911>
closer = 'serhiy.storchaka'
components = ['Windows']
creation = <Date 2011-10-06.20:59:47.493>
creator = 'M..Z.'
dependencies = []
files = ['23330', '23331', '23332', '26644', '26660', '26694', '26695']
hgrepos = []
issue_num = 13119
keywords = ['patch']
message_count = 36.0
messages = ['145039', '145050', '145054', '145064', '145066', '145068', '145310', '167067', '167092', '167193', '167250', '167257', '167269', '167288', '167379', '167382', '167449', '167452', '167454', '167457', '167460', '167465', '167468', '167479', '167481', '167482', '167485', '167486', '167488', '167499', '167500', '167502', '168348', '168349', '168350', '168351']
nosy_count = 10.0
nosy_names = ['loewis', 'georg.brandl', 'ishimoto', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'flox', 'M..Z.', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13119'
versions = ['Python 3.2', 'Python 3.3']

MZ · 2011-10-06T20:59:47Z

In 3.2.2 the newline for print() is \n on Windows, and not \r\n as expected.

In 3.1.4 the newline is \r\n.

OS is Win 7, and tried on both 32 and 64 bit.

Small example with output is attached.

amauryfa · 2011-10-06T23:05:48Z

To people who open the file in their browser: text files are very similar, but newline_3.1.txt has CRLF line endings and newline_3.2.txt has LF line endings.

M.Z, how did you obtain them? did you start a subprocess?

MZ · 2011-10-07T04:47:56Z

Hi Amaury,

The two text files were obtained through redirection in Windows, so I simply ran the newline.py file with:

...> C:\Python31\python.exe newline.py > newline_3.1.txt

...> C:\Python32\python.exe newline.py > newline_3.2.txt

Best regards,
Morten Zilmer

vstinner · 2011-10-07T08:52:32Z

I changed how newlines are handled on Windows to fix an issue with CGI: see the issue bpo-10841.

changeset: 67431:0933c3753a71
user: Victor Stinner <victor.stinner@haypocalc.com>
date: Fri Jan 07 18:47:22 2011 +0000
files: Misc/NEWS Modules/_io/fileio.c Modules/main.c Parser/tokenizer.c
description:
Issue bpo-10841: set binary mode on files; the parser translates newlines

On Windows, set the binary mode on stdin, stdout, stderr and all
io.FileIO objects (to not translate newlines, \r\n <=> \n). The Python parser
translates newlines (\r\n => \n).

vstinner · 2011-10-07T09:01:18Z

print() uses PyFile_WriteString("\n", file) by default (if the end argument is not set) to write the newline. TextIOWrapper.write("\n") replaces "\n" by TextIOWrapper._writenl.

On Windows, stdin, stdout and stderr are creates using TextIOWrapper(..., newline=None). In this case, TextIOWrapper._writenl is os.linesep and so '\r\n'.

To sum up, print() writes '\n' into sys.stdout, but sys.stdout write b'\r\n' into the file descriptor 1 which is a binary file (ie. the underlying OS file doesn't translate newlines).

If the output is redirected (e.g. into a file), TextIOWrapper is created with line_buffering=False.

You may try to force line_buffering=True when the output is redirected.

amauryfa · 2011-10-07T09:32:45Z

If the output is redirected (e.g. into a file),
TextIOWrapper is created with line_buffering=False.
How does this affect the \r\n translation?

MZ · 2011-10-10T17:51:50Z

Just to make it clear: I have not observed any problems on the Windows terminal (cmd) with \n newline, but at least Notepad does not break lines correctly if only \n is used.

atsuoishimoto · 2012-08-01T01:45:03Z

I found 'more' command in Windows7 requires \r\n.

Python 2.7.3:

C:\>python -c "for i in range(5):print(i)"|more
0
1
2
3
4

Python 3.3(trunk):

c:\src\cpython\PCbuild>python -c "for i in range(5):print(i)"|more
?????

vstinner · 2012-08-01T07:20:00Z

On Windows, stdin, stdout and stderr are creates using TextIOWrapper(..., newline=None).
In this case, TextIOWrapper._writenl is os.linesep and so '\r\n'.

Oh, I was wrong: stdin is created with newline=None, but stdout and stderr are created with newline="\n" and so "\n" is not translated to "\r\n".

I checked in Python 2.7: print("abc") and sys.stdout.write("abc\n") writes b"abc\r\n" into the output file (when the output is redirected), but sys.stdout.write("abc\r\n") writes b"abc\r\r\n". Python 3.3 should do the same: \r\n is preferred on Windows (ex: notepad doesn't support UNIX line ending, \n).

Attached patch changes line ending for stdout and stderr on Windows: translate "\n" to "\r\n".

It would be nice to fix this before Python 3.3 final.

atsuoishimoto · 2012-08-02T05:20:09Z

Test for this issue. Tested on Windows7, Ubuntu linux 12.04.

I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.

But in Python2.7, "print >>sys.stderr, 1" also returns '1',
so this might not be a problem.

vstinner · 2012-08-02T20:16:44Z

I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.

I suppose that you mean "returns '1\n' instead of '1'". This is a
major change between Python 2 and Python 3. Use print(1, end=' ') if
you want the same behaviour. See:
http://docs.python.org/dev/whatsnew/3.0.html#print-is-a-function

You can also use the print() as a function in Python 2 using "from
__future__ import print_function":
http://docs.python.org/dev/whatsnew/2.6.html#pep-3105-print-as-a-function

I never liked "print expr," because it looks like a typo or an ugly
hack. It's easy to add a comma by mistake.

vstinner · 2012-08-02T21:11:04Z

It would be nice to fix this before Python 3.3 final.

@georg: So, what do you think?

pitrou · 2012-08-02T22:33:39Z

About the patch: why wouldn't you use newline = NULL in both cases?

atsuoishimoto · 2012-08-03T04:44:28Z

On Fri, Aug 3, 2012 at 5:16 AM, STINNER Victor <report@bugs.python.org> wrote:

> I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.

I suppose that you mean "returns '1\n' instead of '1'".

No, sorry for my lame wording.

In the test I submitted, printing to stdout with

"print(1, file=sys.stdout);print(2, file=sys.stdout)"

outputs

"1\r\n2\r\n"

but printing to stderr with

"print(1, file=sys.stderr);print(2, file=sys.stderr)"

outputs

"1\r\n2"   <- no '\r\n' at the end

I wondered why, but this is not specific to Python 3.
With Python 2.7

print >>sys.stderr, 1

doesn't output '\r\n' at the end also. So I think this may
not be a bug.

python-dev · 2012-08-03T23:31:57Z

New changeset c55dbb84f3b4 by Victor Stinner in branch 'default':
Close bpo-13119: use "\r\n" newline for sys.stdout/err on Windows
http://hg.python.org/cpython/rev/c55dbb84f3b4

python-dev · 2012-08-03T23:42:20Z

New changeset 09408b990ca5 by Victor Stinner in branch '3.2':
Close bpo-13119: use "\r\n" newline for sys.stdout/err on Windows
http://hg.python.org/cpython/rev/09408b990ca5

pitrou · 2012-08-04T22:11:14Z

Windows buildbots now show failures in the test suite.

python-dev · 2012-08-04T22:29:44Z

New changeset 4efad7fba42a by Antoine Pitrou in branch '3.2':
Fix test_sys under Windows (issue bpo-13119)
http://hg.python.org/cpython/rev/4efad7fba42a

New changeset e4a87f0253e9 by Antoine Pitrou in branch 'default':
Merge universal newlines-related fixes (issue bpo-13119)
http://hg.python.org/cpython/rev/e4a87f0253e9

python-dev · 2012-08-04T22:35:53Z

New changeset f8e435d6a801 by Antoine Pitrou in branch 'default':
Fix test_venv to work with universal newlines (issue bpo-13119)
http://hg.python.org/cpython/rev/f8e435d6a801

pitrou · 2012-08-04T22:46:33Z

test_httpservers still fails, it's the CGI tests...

atsuoishimoto · 2012-08-04T23:41:41Z

Fix for test_httpservers

atsuoishimoto · 2012-08-05T00:40:50Z

Sorry, please ignore the patch 'issue13119_httpserver.patch' I posted above.

Behavior of "-u" commandline option in Python3.3 is differ than in Python 2.

We should not convert newline characters if "-u" specified? I'll investigate more.

atsuoishimoto · 2012-08-05T01:02:00Z

We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.

Added a patch to fix error in test_httpservers.

pitrou · 2012-08-05T10:26:05Z

We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.

Why that? What do universal newlines have to do with buffering?

loewis · 2012-08-05T10:44:33Z

I wonder why this is a release blocker. It's a bug in Python 3.2, so why should it block the release of 3.3 (it's not a regression).

If no complete solution is coming up, I recommend to revert all changes on this issue, and reconsider after the 3.3 release.

pitrou · 2012-08-05T10:49:09Z

I wonder why this is a release blocker. It's a bug in Python 3.2, so
why should it block the release of 3.3 (it's not a regression).

It's a blocker because the fix broke a couple of tests.
And it's also a regression from 3.1 and 2.7.

loewis · 2012-08-05T11:03:28Z

It's a blocker because the fix broke a couple of tests.

That cannot possibly be the explanation why haypo declared
it a blocker on 2012-08-01; the fix was only applied on
2012-08-04.

But I agree that it should block the release now; hence I
propose to roll back the entire set of changes and revert
to the 3.2 state.

And it's also a regression from 3.1 and 2.7.

Since 3.2 was released with this behavior, this bug cannot manage
to block 3.3.

atsuoishimoto · 2012-08-05T11:16:13Z

Antoine Pitrou added the comment:

> We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.

Why that? What do universal newlines have to do with buffering?

Man page of Python says

-u Force stdin, stdout and stderr to be totally unbuffered. On
systems where it matters, also put stdin, stdout and stderr in
binary mode.

test_httpservers depends on this behavior, but was implemented as documented in Python 3.

pitrou · 2012-08-05T11:21:18Z

Man page of Python says

-u Force stdin, stdout and stderr to be totally unbuffered. On
systems where it matters, also put stdin, stdout and stderr in
binary mode.

I don't know which version it is, but current 3.3 says:

“Force the binary I/O layers of stdin, stdout and stderr to be
unbuffered. The text I/O layer will still be line-buffered.”

test_httpservers depends on this behavior, but was implemented as
documented in Python 3.

I would argue that test_httpservers is wrong, since it uses print() in a
CGI script where sys.stdout.buffer.write() should really be used.

python-dev · 2012-08-05T12:56:33Z

New changeset bc4fdb758b8c by Antoine Pitrou in branch '3.2':
Fix CGI tests to take into account the platform's line ending (issue bpo-13119)
http://hg.python.org/cpython/rev/bc4fdb758b8c

New changeset ee185c6b2880 by Antoine Pitrou in branch 'default':
Fix CGI tests to take into account the platform's line ending (issue bpo-13119)
http://hg.python.org/cpython/rev/ee185c6b2880

vstinner · 2012-08-05T13:22:34Z

Buildbots are happy, the issue can be closed again. Thanks Antoine.

atsuoishimoto · 2012-08-05T13:28:06Z

I don't know which version it is, but current 3.3 says:

Ah, sorry, I thought I was reading latest Man page.

serhiy-storchaka · 2012-08-16T06:28:06Z

" newline is '' or '\n', no translation takes place. If newline is any\n"

Non-escaped "\n".

birkenfeld · 2012-08-16T06:33:05Z

Would be nice to be a bit more specific *where* that line comes from.

serhiy-storchaka · 2012-08-16T06:54:33Z

Would be nice to be a bit more specific *where* that line comes from.

Modules/_io/textio.c, changesets 243ad1a6f638 and 083776adcacc.

serhiy-storchaka · 2012-08-16T06:58:04Z

Oops, I got the wrong issue, sorry.

MZ mannequin added OS-windows type-bug An unexpected behavior, bug, or error labels Oct 6, 2011

vstinner added the release-blocker label Aug 1, 2012

python-dev mannequin closed this as completed Aug 3, 2012

pitrou reopened this Aug 4, 2012

vstinner closed this as completed Aug 5, 2012

serhiy-storchaka reopened this Aug 16, 2012

serhiy-storchaka closed this as completed Aug 16, 2012

ezio-melotti transferred this issue from another repository Apr 10, 2022

UnderCooled mentioned this issue Jan 12, 2024

Some code tools return "\r\n" on Windows. nvimdev/guard.nvim#130

Closed

Newline for print() is \n on Windows, and not \r\n as expected #57328

Newline for print() is \n on Windows, and not \r\n as expected #57328

Comments

MZ mannequin commented Oct 6, 2011

MZ mannequin commented Oct 6, 2011

amauryfa commented Oct 6, 2011

MZ mannequin commented Oct 7, 2011

vstinner commented Oct 7, 2011

vstinner commented Oct 7, 2011

amauryfa commented Oct 7, 2011

MZ mannequin commented Oct 10, 2011

atsuoishimoto mannequin commented Aug 1, 2012

vstinner commented Aug 1, 2012

atsuoishimoto mannequin commented Aug 2, 2012

vstinner commented Aug 2, 2012

vstinner commented Aug 2, 2012

pitrou commented Aug 2, 2012

atsuoishimoto mannequin commented Aug 3, 2012

python-dev mannequin commented Aug 3, 2012

python-dev mannequin commented Aug 3, 2012

pitrou commented Aug 4, 2012

python-dev mannequin commented Aug 4, 2012

python-dev mannequin commented Aug 4, 2012

pitrou commented Aug 4, 2012

atsuoishimoto mannequin commented Aug 4, 2012

atsuoishimoto mannequin commented Aug 5, 2012

atsuoishimoto mannequin commented Aug 5, 2012

pitrou commented Aug 5, 2012

loewis mannequin commented Aug 5, 2012

pitrou commented Aug 5, 2012

loewis mannequin commented Aug 5, 2012

atsuoishimoto mannequin commented Aug 5, 2012

pitrou commented Aug 5, 2012

python-dev mannequin commented Aug 5, 2012

vstinner commented Aug 5, 2012

atsuoishimoto mannequin commented Aug 5, 2012

serhiy-storchaka commented Aug 16, 2012

birkenfeld commented Aug 16, 2012

serhiy-storchaka commented Aug 16, 2012

serhiy-storchaka commented Aug 16, 2012