classification
Title: reject unicode in zlib
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: ebfe, flox, lemburg, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2008-12-27 12:58 by vstinner, last changed 2010-01-10 02:31 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
zlib_bytes.patch vstinner, 2008-12-27 12:58
issue4757_zlib_bytes_v2.diff flox, 2009-12-14 17:04 Patch, apply to py3k
Messages (15)
msg78356 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-27 12:58
Python 2.x allows to encode any byte string (str) and ASCII unicode 
string (unicode):

$ python
Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:40)
>>> import zlib
>>> zlib.compress('abc')
"x\x9cKLJ\x06\x00\x02M\x01'"
>>> zlib.compress(u'abc')
"x\x9cKLJ\x06\x00\x02M\x01'"
>>> zlib.compress(u'abc\xe9')
...
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' ...

I'm not sure that this behaviour was really wanted become the 
decompress operation is not symetric (the result type is always byte 
string):

$ python
Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:40)
>>> import zlib
>>> zlib.decompress("x\x9cKLJ\x06\x00\x02M\x01'")
'abc'

---

Python 3.0 accepts any string: bytes or characters. But decompress 
always produce bytes string:

$ ./python
Python 3.1a0 (py3k:67926M, Dec 26 2008, 23:59:07)
>>> import zlib
>>> zlib.compress(b'abc')
b"x\x9cKLJ\x06\x00\x02M\x01'"
>>> zlib.compress('abc')
b"x\x9cKLJ\x06\x00\x02M\x01'"
>>> zlib.compress('abc\xe9')
b'x\x9cKLJ>\xbc\x12\x00\x06\xca\x02\x93'
>>> zlib.compress('abc\xe9'.encode('utf-8'))
b'x\x9cKLJ>\xbc\x12\x00\x06\xca\x02\x93'
>>> zlib.decompress(b'x\x9cKLJ>\xbc\x12\x00\x06\xca\x02\x93')
b'abc\xc3\xa9'

The most strange operation is the decompression of an unicode string:

$ ./python
>>> zlib.decompress('x\x9cKLJ>\xbc\x12\x00\x06\xca\x02\x93')
...
zlib.error: Error -3 while decompressing data: incorrect header check

---

I propose to change zlib API to reject unicode string and use explicit 
conversion to/from bytes. Functions/methods:
 - compress(bytes, ...)
 - decompress(bytes, ...)
 - <compress object>.compress(bytes, ...)
 - <decompress object>.decompress(bytes, ...)
 - crc32(bytes, value=0)
 - adler(bytes, value=1)

Note: binascii.crc32() already rejects unicode string.

The behaviour may kept in Python 3.0.x and only changed in Python 3.1.
msg78357 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-27 13:00
See also issue #4738 (better threads support in zlib).
msg78360 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-12-27 13:13
On 2008-12-27 13:58, STINNER Victor wrote:
> Python 2.x allows to encode any byte string (str) and ASCII unicode 
> string (unicode):
> 
> $ python
> Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:40)
>>>> import zlib
>>>> zlib.compress('abc')
> "x\x9cKLJ\x06\x00\x02M\x01'"
>>>> zlib.compress(u'abc')
> "x\x9cKLJ\x06\x00\x02M\x01'"
>>>> zlib.compress(u'abc\xe9')
> ...
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' ...
> 
> I'm not sure that this behaviour was really wanted become the 
> decompress operation is not symetric (the result type is always byte 
> string):
> 
> $ python
> Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:40)
>>>> import zlib
>>>> zlib.decompress("x\x9cKLJ\x06\x00\x02M\x01'")
> 'abc'
> 

I don't see a problem with this. The fact that Python 2.x also
accepts Unicode ASCII strings where strings are normally expected
is intended to help with the migration to Unicode, so the above
is expected.

zlib itself doesn't care about whether the data to be encoded
is text or bytes.

In Python 3.x, it's probably better to use bytes throughout the
API.
msg78365 - (view) Author: Lukas Lueg (ebfe) Date: 2008-12-27 13:58
I don't think Python 2.x should be changed - but 3.0 or 3.1 should be:

 - Characters don't mean a thing in zlib-land, all operations are based
on bytes and their (implicit) default encoding. This behaviour is hidden
and somewhat violates the rule of least surprise.
 - type(zlib.decompress(zlib.compress('abc'))) == bytes anyway
 - Changing from s* to y* forces the programmer to use .encode() on his
strings (e.g. zlib.compress('abc'.encode()) which very clearly shows
what's happening. If you want to compress and decompress Python3
strings, you *must* share the same character encoding; think of
zlib.compress('hôńè') and str(zlib.decompress(x)) with different locales.
 - Other modules (hashlib comes to my mind...) already reject Unicode
objects for the same argument.
msg79090 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-04 22:51
> The fact that Python 2.x also accepts Unicode ASCII strings 
> where strings are normally expected is intended to help with 
> the migration to Unicode

I hate this behaviour. It doesn't help migration, it's the opposite! Sometimes 
it works (ASCII), and somtimes it fails (just one non-ASCII character). And 
then we will read "Unicode sucks!" because people doesn't understand the 
error.

> In Python 3.x, it's probably better to use bytes throughout the
> API.

I propose to reject unicode in Python 3.x and display a warning for Python 
2.x. A warning to prepare the migration... not to Unicode, but to Python3 ;-)
msg79110 - (view) Author: Lukas Lueg (ebfe) Date: 2009-01-05 06:44
The current behaviour may help the majority by ignorance and cause weird
errors for others. We tell people that Python distincts between Text and
Data but actually treat it all the same by implicit encoding.

Modules that only operate on Bytes should reject Unicode-objects in
Python3; it's a matter of 3 lines to display a warning in Python 2.
Those modules that usually operate on Text but have single functions
that operate on Bytes should display a warning but not enforce explicit
encoding.

Also see #4821 and #4818 where unicode already got rejected by the
openssl-driven classes but silently accepted by the build-in ones.
msg79124 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2009-01-05 10:44
On 2009-01-04 23:51, STINNER Victor wrote:
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> The fact that Python 2.x also accepts Unicode ASCII strings 
>> where strings are normally expected is intended to help with 
>> the migration to Unicode
> 
> I hate this behaviour. It doesn't help migration, it's the opposite! Sometimes 
> it works (ASCII), and somtimes it fails (just one non-ASCII character). And 
> then we will read "Unicode sucks!" because people doesn't understand the 
> error.

Well, that's your opinion.

The feature was added to get people
work with Unicode at all, since otherwise we would have had to do
all the Unicode porting we're doing now for Python 3 at the time
Unicode was introduced - which was in version Python 1.6, eight years
ago.

At the time the Python community was a lot smaller and there wasn't
all that much interest in Unicode anyway - the Unicode support I wrote
for Python 1.6 was partially financed by HP which needed it for an
application they had written in Python.

See the introduction in PEP 100 for the motivation behind the design
decisions:

http://www.python.org/dev/peps/pep-0100/

>> In Python 3.x, it's probably better to use bytes throughout the
>> API.
> 
> I propose to reject unicode in Python 3.x and display a warning for Python 
> 2.x. A warning to prepare the migration... not to Unicode, but to Python3 ;-)

Fair enough.
msg79225 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-06 01:38
> > I propose to reject unicode in Python 3.x and display a warning for
> > Python 2.x. A warning to prepare the migration... not to Unicode, but to
> > Python3 ;-)
>
> Fair enough.

The patch for Python 3.x is already attached to this issue. We might only 
apply this one and leave Python 2.x unchanged. Can someone review the patch?
msg96376 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2009-12-14 13:55
Definitely, zlib.compress should raise a TypeError (like bz2 does).

>>> import bz2, zlib
>>> bz2.compress('abc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: argument 1 must be bytes or buffer, not str
>>> zlib.compress('abc')
b"x\x9cKLJ\x06\x00\x02M\x01'"

Someone can review the patch and merge it?
msg96378 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-14 15:05
The patch lacks a test that TypeError is raised on unicode input,
otherwise it's fine.
msg96382 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2009-12-14 16:35
Patch from haypo updated for r76830 .

Additional tests for TypeError, and to check that bytearray objects are
accepted.
msg96383 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-14 16:55
The patch produces a number of errors in test_tarfile, test_distutils,
test_gzip and test_xmlrpc.
msg96384 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2009-12-14 17:04
Fixed.

And some "bytearray" tests improved in test_zlib.
msg96389 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-14 18:24
The patch was committed to py3k and 3.1. Thank you!
msg97491 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-01-10 02:31
> The patch was committed to py3k and 3.1. Thank you!

r76836 and r76838
History
Date User Action Args
2010-01-10 02:31:01vstinnersetmessages: + msg97491
2009-12-14 19:08:39floxsetfiles: - issue4757_zlib_bytes.diff
2009-12-14 18:24:46pitrousetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg96389
2009-12-14 18:00:19pitrousetassignee: pitrou
resolution: accepted
2009-12-14 17:04:28floxsetfiles: + issue4757_zlib_bytes_v2.diff

messages: + msg96384
2009-12-14 16:55:05pitrousetmessages: + msg96383
2009-12-14 16:35:04floxsetfiles: + issue4757_zlib_bytes.diff

messages: + msg96382
2009-12-14 15:05:26pitrousetmessages: + msg96378
2009-12-14 13:55:31floxsetnosy: + flox

messages: + msg96376
versions: + Python 3.2, - Python 3.0
2009-01-06 01:38:09vstinnersetmessages: + msg79225
2009-01-05 10:44:26lemburgsetmessages: + msg79124
2009-01-05 06:44:09ebfesetmessages: + msg79110
2009-01-04 22:51:12vstinnersetmessages: + msg79090
2008-12-27 13:58:22ebfesetmessages: + msg78365
2008-12-27 13:13:13lemburgsetnosy: + lemburg
messages: + msg78360
2008-12-27 13:00:01vstinnersetnosy: + pitrou, ebfe
messages: + msg78357
2008-12-27 12:58:18vstinnercreate