This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Add link to alternatives for bytes-to-bytes codecs
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, doerwalter, ezio.melotti, flox, lemburg, ncoghlan, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-04-25 11:37 by serhiy.storchaka, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
doc_codecs_impl.patch serhiy.storchaka, 2013-05-21 10:20 Patch for 3.x review
doc_codecs_impl-2.7_2.patch serhiy.storchaka, 2013-05-21 14:28 Patch for 2.7 review
Messages (15)
msg187777 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-04-25 11:37
The proposed patch adds link to alternative interfaces for bytes-to-bytes codecs. I.e. base64.b64encode and base64.b64decode for base64_codec.

Patch for 2.7 should mention other functions/modules (due to lack of some of them).
msg189540 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-18 18:47
Any opinions?
msg189578 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-19 11:09
I like this, both because it quite clearly defines the encode and decode directions, and allows notes the more direct entry points if the codec isn't being specified as an input string.

So +1 from me.
msg189579 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2013-05-19 11:31
Not a bad idea. More information is always better when it comes to
documentation :-)
msg189740 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-21 10:20
> Not a bad idea.

How about implementation? Here is updated patches for 3.x and 2.7. Note that in 2.7 I split codecs table as in 3.x.
msg189743 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-21 11:14
I like the idea of splitting the table in 2.7 rather than using a result type column. However, the two intro paragraphs need a bit of work. How does the following sound:

1. Create a new subheading at the same level as the current "Standard Encodings" heading: "Python Specific Encodings"

2. Split out rot-13 to its own table in Python 2.7 as well

3. Under the new subheading, have the following text introducing the tables:

----
A number of predefined codecs are specific to Python, so their codec names have no meaning outside Python. These are listed in the tables below based on the expected input and output types (note that while text encodings are the most common use case for codecs, the underlying codec infrastructure supports arbitrary data transforms rather than just text encodings).  For asymmetric codecs, the stated purpose describes the encoding direction.

The following codecs provide text-to-binary encoding and binary-to-text decoding, similar to the Unicode text encodings.
----
The following codecs provide binary-to-binary encoding and decoding.
----
The following codecs provide text-to-text encoding and decoding.
----
msg189747 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-21 12:39
> However, the two intro paragraphs need a bit of work.

Yes, it's a help which I needed. Thank you.

However your wording is not entirely correct. In 2.7 binary-to-binary codecs and rot-13 works with Unicode strings (only ascii-compatible) as with bytes strings.

>>> u'Python'.encode('base64')
'UHl0aG9u\n'
>>> u'UHl0aG9u'.decode('base64')
'Python'
>>> u'Python\u20ac'.encode('base64')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-2.7/Lib/encodings/base64_codec.py", line 24, in base64_encode
    output = base64.encodestring(input)
  File "/home/serhiy/py/cpython-2.7/Lib/base64.py", line 315, in encodestring
    pieces.append(binascii.b2a_base64(chunk))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 6: ordinal not in range(128)

Rot-13 works as common text-to-binary encoding (encode returns str, decode returns unicode).

>>> u'Python'.encode('rot13')
'Clguba'
>>> u'Python'.decode('rot13')
u'Clguba'
>>> 'Python'.encode('rot13')
'Clguba'
>>> 'Python'.decode('rot13')
u'Clguba'
>>> u'Python\u20ac'.encode('rot13')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-2.7/Lib/encodings/rot_13.py", line 17, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u20ac' in position 6: character maps to <undefined>
>>> u'Python\u20ac'.decode('rot13')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-2.7/Lib/encodings/rot_13.py", line 20, in decode
    return codecs.charmap_decode(input,errors,decoding_map)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 6: ordinal not in range(128)
msg189749 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-21 13:12
While the Python 2 text model was almost certainly a necessary transition step to full unicode support, it is things like this that highlight how fundamentally broken implicit conversion turned out to be at a conceptual level :P

Perhaps the following would work for 2.7 then (with rot-13 in the first table), with footnotes added to cover the quirks of the implicit type conversions between str and unicode:

----
A number of predefined codecs are specific to Python, so their codec names have no meaning outside Python. These are listed in the tables below based on the expected input and output types (note that while text encodings are the most common use case for codecs, the underlying codec infrastructure supports arbitrary data transforms rather than just text encodings).  For asymmetric codecs, the stated purpose describes the encoding direction.

The following codecs provide unicode-to-str encoding [#1] and str-to-unicode decoding [#2], similar to the Unicode text encodings.
----
The following codecs provide str-to-str encoding and decoding [#2].
----

.. [#1] str objects are also accepted as input in place of unicode objects. They are implicitly converted to unicode by decoding them using the default encoding. If this conversion fails, it may lead to encoding operations raising :exc:`UnicodeDecodeError`.

.. [#2] unicode objects are also accepted as input in place of str objects. They are implicitly converted to str by encoding them using the default encoding. If this conversion fails, it may lead to decoding operations raising :exc:`UnicodeEncodeError`.
msg189761 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-21 14:28
Thank you Nick. Here is an updated patch for 2.7.
msg189797 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-22 08:11
Thanks Serhiy, that version looks great.
msg189811 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-05-22 12:36
New changeset 85c04fdaa404 by Serhiy Storchaka in branch '2.7':
Issue #17844: Refactor a documentation of Python specific encodings.
http://hg.python.org/cpython/rev/85c04fdaa404

New changeset 039dc6dd2bc0 by Serhiy Storchaka in branch '3.3':
Issue #17844: Add links to encoders and decoders for bytes-to-bytes codecs.
http://hg.python.org/cpython/rev/039dc6dd2bc0

New changeset 9afdd88fe33a by Serhiy Storchaka in branch 'default':
Issue #17844: Add links to encoders and decoders for bytes-to-bytes codecs.
http://hg.python.org/cpython/rev/9afdd88fe33a
msg189812 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-22 12:40
Thank you Nick. It's mainly your patch.

Do you want to foreport your changes (a "Python Specific Encodings" subheading and followed paragraph) to 3.x?
msg189821 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-22 14:50
That sounds like a good idea. Yay for not needing those arcane footnotes, though :)
msg189856 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-05-23 10:25
New changeset 85e8414060b4 by Nick Coghlan in branch '3.3':
Issue 17844: Clarify meaning of different codec tables
http://hg.python.org/cpython/rev/85e8414060b4

New changeset 801567d6302c by Nick Coghlan in branch 'default':
Merge issue 17844 from 3.3
http://hg.python.org/cpython/rev/801567d6302c
msg189857 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-23 10:26
Thanks for initiating this Serhiy :)
History
Date User Action Args
2022-04-11 14:57:44adminsetgithub: 62044
2013-05-23 10:26:34ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg189857

stage: patch review -> resolved
2013-05-23 10:25:22python-devsetmessages: + msg189856
2013-05-22 15:01:26ezio.melottisetnosy: + ezio.melotti
2013-05-22 14:50:57ncoghlansetmessages: + msg189821
2013-05-22 12:40:01serhiy.storchakasetmessages: + msg189812
2013-05-22 12:36:17python-devsetnosy: + python-dev
messages: + msg189811
2013-05-22 08:11:34ncoghlansetmessages: + msg189797
2013-05-21 14:32:19serhiy.storchakasetfiles: - doc_codecs_impl-2.7.patch
2013-05-21 14:30:16floxsetnosy: + flox
2013-05-21 14:28:18serhiy.storchakasetfiles: + doc_codecs_impl-2.7_2.patch

messages: + msg189761
2013-05-21 13:12:11ncoghlansetmessages: + msg189749
2013-05-21 12:39:04serhiy.storchakasetmessages: + msg189747
2013-05-21 11:14:27ncoghlansetmessages: + msg189743
2013-05-21 10:21:20serhiy.storchakasetfiles: - doc_codecs_impl.patch
2013-05-21 10:20:49serhiy.storchakasetfiles: + doc_codecs_impl-2.7.patch
2013-05-21 10:20:00serhiy.storchakasetfiles: + doc_codecs_impl.patch

messages: + msg189740
2013-05-19 11:31:36lemburgsetmessages: + msg189579
2013-05-19 11:09:13ncoghlansetmessages: + msg189578
2013-05-18 18:47:28serhiy.storchakasetmessages: + msg189540
2013-04-25 11:43:30serhiy.storchakalinkissue7475 dependencies
2013-04-25 11:37:22serhiy.storchakacreate