classification
Title: patch: steer people away from codecs.open
Type: behavior Stage:
Components: Documentation Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Frank.van.Dijk, docs@python, doerwalter, lemburg, martin.panter, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2014-08-03 13:17 by Frank.van.Dijk, last changed 2015-01-02 20:16 by martin.panter.

Files
File name Uploaded Description Edit
codecsopen2.patch Frank.van.Dijk, 2014-08-03 13:17 patch against 2.7 review
codecsopen3.patch Frank.van.Dijk, 2014-08-03 13:17 patch against 3.5; applies to 3.4 too review
codecsopen2a.patch Frank.van.Dijk, 2014-08-05 19:41 toned down variant of codecsopen2.patch in response to http://bugs.python.org/issue22128#msg224659 review
codecsopen3a.patch Frank.van.Dijk, 2014-08-05 19:42 toned down variant of codecsopen3.patch in response to http://bugs.python.org/issue22128#msg224659 review
Messages (6)
msg224632 - (view) Author: Frank van Dijk (Frank.van.Dijk) * Date: 2014-08-03 13:17
stackoverflow.com has a zillion answers recommending the use of codecs.open() as a unicode capable drop in replacement for open(). This probably means that there is still a lot of code being written that uses codecs.open(). That's bad thing because of codecs.open()'s lack of newline conversion. A lot of that code will 
- have compatibility issues when it is moved between unix and windows
- silently break text files on windows, leading to issues further downstream (confusing other tools, messing up revision control histories)

The problem has been fixed with io.open() in 2.x and open() in 3.x. Unfortunately the 2.7 unicode HOWTO still recommends the use of codecs.open(). The 2.7 and the 3.x documentation of codecs.open() doesn't refer the reader to better alternatives.

The attached patches fix that.

The only downside I see is that newly written code that uses the better alternatives would be incompatible with 2.5 and older. However croaking on a small minority of systems is better than silently disrupting workflows, causing platform incompatibilities, and inviting flaky workarounds.

The 2.7 patch makes the unicode HOWTO recommend io.open() instead of codecs.open(). Both patches change the codecs.open() documentation to refer to io.open() or (on 3.x) open().

Additionally I removed the "data loss" explanation from codecs.open()'s note about its lack of newline conversion. It is not particularly helpful information and it is not entirely correct (data loss could also have been avoided by doing newline conversion before encoding and after decoding)
msg224657 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-03 20:35
See also my PEP 400:
http://legacy.python.org/dev/peps/pep-0400/
msg224659 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-08-03 20:45
Pointing people to io.open() as alternative to codecs.open() is a good idea, but that doesn't make codecs.open() less useful.

The reason why codecs.open() uses binary mode is to avoid issues with automatic newline conversion getting in the way of the file's encoding. Think of e.g. UTF-16 encoded files that use newlines.

Note that codecs allow handling newlines on a line-by-line bases via the .readline() keepends parameter, so issues with Windows vs. Unix can be worked around explicitly. Since default is to keep line ends, no data loss occurs and application code can deal with line ends as it sees fit.

As it stands, I'm -1 on this patch, but would be +1 on mentioning io.open() as alternative to codecs.open() with a slightly different approach to line ends.

I don't think it's useful to tell people:
* use codecs.open() on Python 2.4, 2.5, 2.6
* use io.open() on Python 2.7 (io is too slow on 2.6 to be a real alternative to codecs.open())
* use open() on Python 3.4+

codecs.open() works the same across all these Python versions.
msg224778 - (view) Author: Frank van Dijk (Frank.van.Dijk) * Date: 2014-08-04 21:51
> Marc-Andre Lemburg added the comment:
> 
> Pointing people to io.open() as alternative to codecs.open() is a good idea, but that doesn't make codecs.open() less useful.
> 
> The reason why codecs.open() uses binary mode is to avoid issues with automatic newline conversion getting in the way of the file's encoding. Think of e.g. UTF-16 encoded files that use newlines.

disabling text mode on the underlying file handle to keep a UTF-16 code unit like 0x010a from getting mangled works, but no newline conversion is a high price to pay. Newline conversion should (conceptually) be done before encoding and after decoding. io.open() does it right.

> 
> Note that codecs allow handling newlines on a line-by-line bases via the .readline() keepends parameter, so issues with Windows vs. Unix can be worked around explicitly. Since default is to keep line ends, no data loss occurs and application code can deal with line ends as it sees fit.

Trouble is, your average python coder won't do exhaustive research on the pros and cons of the various options for I/O available and the pros and cons of dealing with platform differences at the application level. They'll just use the open() builtin, then realize they need utf-8 output or whatever, google "python write utf-8" or browse the unicode HOWTO, see a very familiar looking API and assume it'll behave just like open()

> 
> As it stands, I'm -1 on this patch, but would be +1 on mentioning io.open() as alternative to codecs.open() with a slightly different approach to line ends.

What would that mean concretely ? Undoing the change to the unicode HOWTO and instead adding a remark along the lines of "The codecs.open() function does not have the automatic newline conversion features that the builtin open() function provides to make reading and writing text files platform independent. If you need automatic newline conversion for the Unicode data you read and write, consider using io.open() instead." ?

I could live with that.

> 
> I don't think it's useful to tell people:
> * use codecs.open() on Python 2.4, 2.5, 2.6
> * use io.open() on Python 2.7 (io is too slow on 2.6 to be a real alternative to codecs.open())
> * use open() on Python 3.4+

The unicode HOWTO already recommends open() on all 3.x versions of the documentation at docs.python.org.

If you run 2.4 and 2.5 and you're adding new python software to your ancient system without upgrading python itself the only thing that could happen is that you'll get a clear-cut error if that new software imports io.

I can't judge how much of a problem slowness of the io module is in 2.6 or how much 'market share' 2.6 has left, but I'll note that correctness trumps performance. I'll also note that we're not changing any code here, nor will there be a rush of coders racing to get their existing apps and frameworks in line with the new decree.

All we're doing is giving average python programmers a better chance to discover what the drop in replacement for open() is or why that helpful tip found on the interwebs left them with a subtly mangled text file that looks really weird in notepad and makes git complain.

> 
> codecs.open() works the same across all these Python versions.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22128>
> _______________________________________
msg224895 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-08-06 02:43
> I don't think it's useful to tell people:
> * use codecs.open() on Python 2.4, 2.5, 2.6
> * use io.open() on Python 2.7 (io is too slow on 2.6 to be a real alternative to codecs.open())
> * use open() on Python 3.4+

Instead we can tell them to use io.open() on all versions from 2.7 and upwards.
2.6 is dead and won't receive any documentation updates anyway.
msg233326 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-01-02 20:16
Just pointing out there is a patch for Issue 19548 for Python 3 which also adds a pointer to the builtin open() function and updates the codecs.open() caveats. That issue doesn’t touch Python 2 though.
History
Date User Action Args
2015-01-02 20:16:39martin.pantersetnosy: + martin.panter
messages: + msg233326
2014-08-06 02:43:37pitrousetnosy: + pitrou
messages: + msg224895
2014-08-05 19:42:01Frank.van.Dijksetfiles: + codecsopen3a.patch
2014-08-05 19:41:09Frank.van.Dijksetfiles: + codecsopen2a.patch
2014-08-04 21:51:00Frank.van.Dijksetmessages: + msg224778
2014-08-03 20:45:50lemburgsetmessages: + msg224659
2014-08-03 20:35:52vstinnersetmessages: + msg224657
2014-08-03 13:20:16serhiy.storchakasetnosy: + lemburg, doerwalter, vstinner
2014-08-03 13:17:58Frank.van.Dijksetfiles: + codecsopen3.patch
2014-08-03 13:17:18Frank.van.Dijkcreate