classification
Title: fileinput should use stdin.buffer for "rb" mode
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: brandon-rhodes, haypo, josh.r, python-dev, sam.kimbrel, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-03-27 13:42 by brandon-rhodes, last changed 2014-05-14 18:17 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
21075-fileinput-stdin.diff sam.kimbrel, 2014-04-15 16:37 review
Messages (5)
msg214952 - (view) Author: Brandon Rhodes (brandon-rhodes) * Date: 2014-03-27 13:42
In Python 3, fileinput.input() returns str lines whether the data is
coming from stdin or from a list of files on the command line. But if
input(mode='rb') is specified, then its behavior becomes inconsistent:
lines from stdin are delivered as already-decoded strings, but data
from files is delivered (correctly) as bytes.

The solution may be that, if a "b" is anywhere in the mode, then input()
should read from the bytes stdin.buffer data source instead of from
stdin.

Otherwise the "rb" mode is rather useless since you can wind
up getting text from it anyway depending on how you are invoked.
msg215001 - (view) Author: Josh Rosenberg (josh.r) * Date: 2014-03-27 23:26
There is a similar, (unfixed?) bug, #14156, in argparse as well. Seems like a common failing in the move to Python 3; std*.buffer was introduced, but none of the places that use it were updated, so they all became str only.
msg216331 - (view) Author: Sam Kimbrel (sam.kimbrel) * Date: 2014-04-15 16:37
Patch attached that checks for 'b' in self._mode and sets self._file to sys.stdin.buffer as appropriate.
msg218558 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-14 18:12
New changeset 7e640fefc9c1 by Serhiy Storchaka in branch '3.4':
Issue #21075: fileinput.FileInput now reads bytes from standard stream if
http://hg.python.org/cpython/rev/7e640fefc9c1

New changeset 4041d4077a85 by Serhiy Storchaka in branch 'default':
Issue #21075: fileinput.FileInput now reads bytes from standard stream if
http://hg.python.org/cpython/rev/4041d4077a85
msg218559 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-14 18:17
Thank you Sam for your contribution.
History
Date User Action Args
2014-05-14 18:17:41serhiy.storchakasetstatus: open -> closed
versions: + Python 3.3
messages: + msg218559

resolution: fixed
stage: resolved
2014-05-14 18:12:34python-devsetnosy: + python-dev
messages: + msg218558
2014-05-14 17:27:41serhiy.storchakasetassignee: serhiy.storchaka

nosy: + serhiy.storchaka
2014-04-16 06:44:30hayposetnosy: + haypo
2014-04-15 16:37:44sam.kimbrelsetfiles: + 21075-fileinput-stdin.diff

nosy: + sam.kimbrel
messages: + msg216331

keywords: + patch
2014-03-27 23:26:29josh.rsetnosy: + josh.r
messages: + msg215001
2014-03-27 13:42:43brandon-rhodescreate