classification
Title: redirected output - stdout writes newline as \n in windows
Type: behavior Stage:
Components: IO, Windows Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, Jimbofbx, amaury.forgeotdarc, astrobuf, haypo, vinay.sajip
Priority: normal Keywords:

Created on 2011-05-03 21:27 by Jimbofbx, last changed 2014-07-29 20:36 by haypo. This issue is now closed.

Messages (11)
msg135076 - (view) Author: James Hutchison (Jimbofbx) Date: 2011-05-03 21:27
In windows, 64-bit, python *mostly* writes only a \n to stdout even though it's mode is 'w'. However it sometimes writes a \r\n on certain print statements and erratically when I have multiple processes writing to stdout.

Output looks fine in console, in IDLE, and using v.3.1

Example with multiple processes writing to stdout out using the same code: print("TESTCODE"); (note that in windows, the naked \n is ignored):

TESTCODETESTCODE
TESTCODE
TESTCODE
TESTCODETESTCODETESTCODE
TESTCODE

Windows program that calls python and receives its piped output is a C# .NET program.
msg135078 - (view) Author: James Hutchison (Jimbofbx) Date: 2011-05-03 21:28
Sorry there isn't more info but I'm really busy right now

In fact a workaround would be appreciated if known.
msg135081 - (view) Author: James Hutchison (Jimbofbx) Date: 2011-05-03 21:52
Nevermind, I have a workaround that didn't require rewriting all the print statements but its in the C# code not the python code
msg135104 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-05-04 09:29
Do you have a test scipt for us to reproduce the issues?
msg135141 - (view) Author: James Hutchison (Jimbofbx) Date: 2011-05-04 16:18
Yes and no, I can give you a single process single child example that just shows that python 3.2 uses binary output while python 3.1 used system default when piping, but trying to reproduce the multiprocessing output inconsistencies would be... difficult. Unfortunately the software I used to spot the \n, \r\n inconsistency with is proprietary. After creating a sample case in only python I couldn't reproduce the inconsistent \r\n issue in python 3.2 so I cannot say for certain that it wasn't caused by my C# program. I wouldn't worry about the inconsistent endlines for now.

Note that I don't see in the "what's new" documentation it mentioning that 3.2 changed the behavior of piped output. Kind of a big deal.

Sample code to compare 3.1 and 3.2 piped output:

import sys;
import os;
import subprocess;
from time import sleep;

python31loc = r"C:\python31\python.exe";
python32loc = r"C:\python32\python.exe";

myname = "IOPipetest.py";

def main(args):
    if(len(args) == 1):
        # main code
        print("Testing python 3.1 endlines.", end='\r\n');
        output = subprocess.check_output("%s %s -output" % (python31loc, myname));
        print(output);
        print("Testing python 3.2 endlines.", end='\r\n');
        output = subprocess.check_output("%s %s -output" % (python32loc, myname));
        print(output);
        sleep(7);
    else:
        for i in range(4):
            print("TESTING DEFAULT"); # endline is supposed to be automatic
            print("TESTING SLASH-EN\n", end='');
            print("TESTING WINDOW-RETURN\r\n", end='');
        

if __name__ == "__main__":
    main(sys.argv);

--------------------------------------

sample output:

Testing python 3.1 endlines.

b'TESTING DEFAULT\r\nTESTING SLASH-EN\r\nTESTING WINDOW-RETURN\r\r\nTESTING DEFAULT\r\nTESTING SLASH-EN\r\nTESTING WINDOW-RETURN\r\r\nTESTING DEFAULT\r\nTESTING SLASH-EN\r\nTESTING WINDOW-RETURN\r\r\nTESTING DEFAULT\r\nTESTING SLASH-EN\r\nTESTING WINDOW-RETURN\r\r\n'
Testing python 3.2 endlines.

b'TESTING DEFAULT\nTESTING SLASH-EN\nTESTING WINDOW-RETURN\r\nTESTING DEFAULT\nTESTING SLASH-EN\nTESTING WINDOW-RETURN\r\nTESTING DEFAULT\nTESTING SLASH-EN\nTESTING WINDOW-RETURN\r\nTESTING DEFAULT\nTESTING SLASH-EN\nTESTING WINDOW-RETURN\r\n'
msg136118 - (view) Author: James Hutchison (Jimbofbx) Date: 2011-05-16 18:48
I would like to add in windows, "input" now adds a \r at the end which wasn't in 3.1. It doesn't do it in idle. This is using just the regular console window that opens up when you double click. I'm guessing this is related to the issue I saw earlier:

code:
from time import sleep

item = input("Input something\n");
print("%s %i" % (item, len(item)));
sleep(15);

in Idle:
Input something
something
something 9

in console window:
Input something
something
 10ething

ouch
msg136119 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-05-16 18:49
That's already fixed, it'll be in 3.2.1
msg143076 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-27 15:19
So is this now just a documentation issue, about the changed behaviour of pipes in 3.2?
msg150921 - (view) Author: Peter Csapo (astrobuf) Date: 2012-01-09 07:12
I have having the same issues as Jimbofbx. This seems to stem from changes due to issue 10841. All stdio is now opened in binary mode, in consideration that it is the TextIOWrapper's job to do endline translation.

The problem here is that the newline mode = '\n' for the TextIOWrapper created for stdout. ( see pythonrun.c: create_stdio() ). For windows, the newline mode for stdin is already set to null enabling universal newline translation on input, and it should be set to null for newline transation on output as well.

OLD CODE
newline = "\n";
#ifdef MS_WINDOWS
if (!write_mode) {
    /* translate \r\n to \n for sys.stdin on Windows */
    newline = NULL;
}
#endif


FIXED??? CODE
newline = "\n";
#ifdef MS_WINDOWS
/* translate \r\n to \n for sys.stdin on Windows */
/* translate \n to \r\n for sys.stdout on Windows */
newline = NULL;
#endif
msg224245 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-29 20:28
It states in msg136119 that this is already fixed and I've confirmed this in 3.4.1 and 3.5.0a0 so believe this can be closed.
msg224246 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-07-29 20:36
Yes, the fix was probably one of these issues:

 - #10841: "binary stdio"
 - #11272: "input() has trailing carriage return on windows", fixed in Python 3.2.1
 - #11395: "print(s) fails on Windows with long strings", fixed in Python 3.2.1
 - #13119: "Newline for print() is \n on Windows, and not \r\n as expected", will be fixed in Python 3.2.4 and 3.3 (not released yet)

Python 3 now always use standard streams in binary mode on Windows.
History
Date User Action Args
2014-07-29 20:36:55hayposetstatus: open -> closed
resolution: fixed
messages: + msg224246
2014-07-29 20:29:52brian.curtinsetnosy: - brian.curtin
2014-07-29 20:28:36BreamoreBoysetnosy: + BreamoreBoy
messages: + msg224245
2012-01-09 07:12:53astrobufsetnosy: + astrobuf
messages: + msg150921
2011-11-15 19:37:26ezio.melottisetnosy: + haypo
2011-08-27 15:19:43vinay.sajipsetnosy: + vinay.sajip
messages: + msg143076
2011-05-16 18:49:29brian.curtinsetnosy: + brian.curtin
messages: + msg136119
2011-05-16 18:48:56Jimbofbxsetmessages: + msg136118
2011-05-04 16:18:35Jimbofbxsetmessages: + msg135141
2011-05-04 09:29:06amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg135104
2011-05-03 21:52:09Jimbofbxsetmessages: + msg135081
2011-05-03 21:28:02Jimbofbxsetmessages: + msg135078
2011-05-03 21:27:09Jimbofbxcreate