This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author jaraco
Recipients ezio.melotti, jaraco, lemburg, loewis, vstinner
Date 2014-07-11.13:01:26
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1405083686.97.0.237614728685.issue21927@psf.upfronthosting.co.za>
In-reply-to
Content
I've tested it and setting PYTHONIOENCODING='utf-8-sig' starts to get there. It causes Python to consume the BOM on stdin, but it also causes stdout to print a spurious non-printable character in the output:

C:\Users\jaraco> echo foo | ./print-input
foo

There is a non-printable character before foo. I've included it in this message. In Powershell, it's rendered with a square before foo:

□foo

Using PowerShell under ConEmu, it appears as a space:

 foo

In cmd.exe, I see this:

C:\Users\jaraco>python -c "print('foo')"
foo


The space before the 'foo' apparently isn't a space at all.

Indeed, the input is being processed as desired, but the output now is not.

C:\Users\jaraco> python -c "print('bar')"
bar

(the non-printable character appears there too)

If I copy that text to the clipboard, I find that character is actually a \ufeff (zero-width no-break space, aka byte order mark). So by setting the environment variable to use utf-8-sig for input, it simultaneously changes the output to also use utf-8-sig. 

So it appears as if setting the environment variable would work for my purposes except that I only want to alter the input encoding and not the output encoding.

I think my goal is pretty basic - read text from standard input and write text to standard output on the primary shell included with the most popular operating system. I contend that goal should be easily achieved and straightforward on Python out of the box.

What does everyone think of the proposal that Python should simply default to utf-8-sig instead of utf-8 for stdin encoding?
History
Date User Action Args
2014-07-11 13:01:27jaracosetrecipients: + jaraco, lemburg, loewis, vstinner, ezio.melotti
2014-07-11 13:01:26jaracosetmessageid: <1405083686.97.0.237614728685.issue21927@psf.upfronthosting.co.za>
2014-07-11 13:01:26jaracolinkissue21927 messages
2014-07-11 13:01:26jaracocreate