classification
Title: fileinput.hook_encoded has no way to pass arguments to codecs
Type: enhancement Stage: resolved
Components: IO, Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: berker.peksag, curioswati, joseph.hackman, lac, python-dev, r.david.murray, serhiy.storchaka, terry.reedy
Priority: normal Keywords: easy, patch

Created on 2015-12-03 16:41 by lac, last changed 2016-04-27 20:16 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
issue25788.patch joseph.hackman, 2016-02-02 00:00 Patch File review
issue25788-2.patch joseph.hackman, 2016-04-27 00:29 review
issue25788-3.patch joseph.hackman, 2016-04-27 16:10 review
Messages (15)
msg255818 - (view) Author: Laura Creighton (lac) Date: 2015-12-03 16:41
Right now there is no way, aside from writing your own openhook, to
get around the limitation that openhook=fileinput.hook_encoded("utf")
will open things with the default option for codecs.open()
of errors=strict.  Adding a way to pass the errors argument seems
both easy to do and useful.
msg255819 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-03 16:52
It would be easy to add the errors parameter to fileinput.hook_encoded().

Do you want to provide a patch Laura?
msg255824 - (view) Author: Laura Creighton (lac) Date: 2015-12-03 17:33
I haven't made a patch to Python for over 10 years. Before mercurial. :) Where do you start in terms of 'how to submit a patch'?
msg255826 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-12-03 17:37
Mercurial didn't change the patch submission process (we're talking about making such changes on python-workflow now...or rather enhancements as the current process should continue to work).  The only thing that mercurial changed is how you get a copy of the current repo.  After that (which is explained in the devguide), just make your changes, do 'hg diff', and attach the diff file to the issue as usual.
msg255827 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-03 17:40
https://docs.python.org/devguide/#contributing
msg256164 - (view) Author: Swati Jaiswal (curioswati) * Date: 2015-12-10 03:23
I want to work on this issue. @lac, can you please help as I searched but couldn't find the related files. Where can I find the code for this?
msg258072 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-01-12 07:47
fileinput is in Python, hence the source is in Lib.  Online version is
https://hg.python.org/cpython/file/tip/Lib/fileinput.py
https://hg.python.org/cpython/file/tip/Lib/test/test_fileinput.py

Patching openhook itself looks trivial.  A test is the hard part. Laura, can you provide a test case? -- a short (one line?) file that fails with errors=strict and passes with something else?  A new test_errors method should be added to class Test_hook_encoded (the last testCase in the file).  I think the 'test' method would need revision.
msg259347 - (view) Author: Joseph Hackman (joseph.hackman) * Date: 2016-02-02 00:00
I haven't seen OP in over 30 days, so am posting my own patch.

I've added an optional argument that defaults to strict and gets passed along.

I've updated the primary test to verify the argument passing, as well as that things get handled as specified in the documentation at https://docs.python.org/3.5/library/codecs.html

This is off-topic, but is there any way I can submit a patch that allows a similar fix for stdin? Presently there is no way at all to pass malformed unicode through fileinput using stdin that I can find.
msg264179 - (view) Author: Joseph Hackman (joseph.hackman) * Date: 2016-04-25 15:18
Ping.

Just wondering if anyone on the nosy list would be willing to help review my patch. :)
msg264193 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-25 19:05
Added comments on Rietveld (follow the "review" link beside the patch link).
msg264334 - (view) Author: Joseph Hackman (joseph.hackman) * Date: 2016-04-27 00:29
Uploading a new patch to address the issues in previous patch.
msg264351 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-27 05:55
Oh, I forget. Needed updates of the documentation (including the "versionchanged" directive). And would be nice if you add corresponding entities in Doc/whatsnew/3.6.rst, Misc/NEWS and Misc/ACKS. The rest of the patch LGTM.
msg264379 - (view) Author: Joseph Hackman (joseph.hackman) * Date: 2016-04-27 16:10
Updated documentation in fileinput.rst, Doc/whatsnew/3.6.rst, Misc/NEWS and Misc/ACKS.

Thank you so much Serhiy for taking the time to review!
msg264400 - (view) Author: Roundup Robot (python-dev) Date: 2016-04-27 20:14
New changeset 8ab8f5259f09 by Serhiy Storchaka in branch 'default':
Issue #25788: fileinput.hook_encoded() now supports an "errors" argument
https://hg.python.org/cpython/rev/8ab8f5259f09
msg264401 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-27 20:16
Committed with some changes. Thank you for your contribution Joseph.
History
Date User Action Args
2016-04-27 20:16:43serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg264401

stage: patch review -> resolved
2016-04-27 20:14:21python-devsetnosy: + python-dev
messages: + msg264400
2016-04-27 16:10:24joseph.hackmansetfiles: + issue25788-3.patch

messages: + msg264379
2016-04-27 05:55:31serhiy.storchakasetmessages: + msg264351
2016-04-27 00:29:48joseph.hackmansetfiles: + issue25788-2.patch

messages: + msg264334
2016-04-25 19:05:34serhiy.storchakasetassignee: serhiy.storchaka
messages: + msg264193
2016-04-25 15:18:51joseph.hackmansetmessages: + msg264179
2016-02-02 14:25:48berker.peksagsetnosy: + berker.peksag

stage: needs patch -> patch review
2016-02-02 00:00:42joseph.hackmansetfiles: + issue25788.patch

nosy: + joseph.hackman
messages: + msg259347

keywords: + patch
2016-01-12 07:47:17terry.reedysetnosy: + terry.reedy
messages: + msg258072
2015-12-10 03:23:54curioswatisetnosy: + curioswati
messages: + msg256164
2015-12-03 17:40:24serhiy.storchakasettype: enhancement
messages: + msg255827
components: + Library (Lib)
versions: - Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5
2015-12-03 17:37:42r.david.murraysetnosy: + r.david.murray
messages: + msg255826
2015-12-03 17:33:50lacsettype: enhancement -> (no value)
messages: + msg255824
components: - Library (Lib)
versions: + Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5
2015-12-03 16:52:46serhiy.storchakasetversions: - Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5
messages: + msg255819

components: + Library (Lib)
keywords: + easy
type: enhancement
stage: needs patch
2015-12-03 16:41:30laccreate