Author steve.dower
Recipients Arfrever, amaury.forgeotdarc, berwin22, chris.jerdonek, davispuh, eric.araujo, eryksun, mark, martin.panter, mightyiam, ncoghlan, pitrou, python-dev, r.david.murray, segfaulthunter, srid, steve.dower, vstinner
Date 2016-12-03.19:47:19
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1480794439.82.0.906189100836.issue6135@psf.upfronthosting.co.za>
In-reply-to
Content
> looks like only way to specify console's codepage is with
`encoding=os.device_encoding(2)` which will have to be used for 99% of cases

That's true, but it only applies when the subprocess is using the same call (GetConsoleCP) to determine what encoding to use to write to something *other* than the console, which is incorrect. The defaults in this cause ought to be ACP, but it depends entirely on the target application and there is no good "99%" default.

To see this demonstrated, check the difference between these two commands (with 3.5 or earlier):

python -c "import sys; print(sys.stdin.encoding)"
python -c "import sys; print(sys.stdin.encoding)" < textfile.txt

When stdin is redirected, the default encoding is not the console encoding (unless it happens to match your ACP). And ACP is not the correct encoding for a file anyway - it just happens to be the default for TextIOWrapper - so to do it properly you need to detect that it's not a console (sys.stdin.isatty()) and then use the raw stream rather than the default wrapper. (Or you specify that your program requires ACP files, or that PYTHONIOENCODING should be set, or that it requires any other encoding and you explicitly switch to that one.)

In short, character encoding is hard, and the only way to get it right is for the developer to be aware of it. No good defaults exist.
History
Date User Action Args
2016-12-03 19:47:19steve.dowersetrecipients: + steve.dower, amaury.forgeotdarc, ncoghlan, pitrou, vstinner, mark, eric.araujo, segfaulthunter, Arfrever, r.david.murray, srid, mightyiam, chris.jerdonek, python-dev, martin.panter, eryksun, berwin22, davispuh
2016-12-03 19:47:19steve.dowersetmessageid: <1480794439.82.0.906189100836.issue6135@psf.upfronthosting.co.za>
2016-12-03 19:47:19steve.dowerlinkissue6135 messages
2016-12-03 19:47:19steve.dowercreate