Title: pickle error in python3.0rc1
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.0
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Georg, benjamin.peterson
Priority: normal Keywords:

Created on 2008-09-18 20:58 by Georg, last changed 2008-09-18 21:04 by benjamin.peterson. This issue is now closed.

Messages (2)
msg73399 - (view) Author: Georg Grafendorfer (Georg) Date: 2008-09-18 20:58
Hi all,
I compiled Python3.0rc1 with the usual ./configure make make test make
install on my Athlon XP 1800 (32 bit), using Debian Etch as OS, the
following works on Python2.4 (default in Debian Etch), but not with

>>> import pickle
>>> d = {'ID':345, 'AD':'Hallo'}
>>> f = open('test.hhh', 'wb')
>>> pickle.dump(d,f,2)
>>> f.close()
>>> f = open('test.hhh', 'r')
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.0/", line 1325, in load
    return Unpickler(file, encoding=encoding, errors=errors).load()
  File "/usr/local/lib/python3.0/", line 1728, in read
    eof = not self._read_chunk()
  File "/usr/local/lib/python3.0/", line 1557, in _read_chunk
    self._set_decoded_chars(self._decoder.decode(input_chunk, eof))
  File "/usr/local/lib/python3.0/", line 1294, in decode
    output = self.decoder.decode(input, final=final)
  File "/usr/local/lib/python3.0/", line 300, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0:
unexpected code byte

the same if you specifiy protocol number 3, it works also in
Python3.0rc1 if you specifiy 'rb' instead of 'r' as file opening method,
but according to the Python library reference it should work also with
'r'.  How should one know with which protocol the object was pickled?

Thanks very much, 

msg73400 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-09-18 21:04
Pickled files should *always* be used in binary mode.

The 3.0 pickle documentation states this:

"Be sure to always open pickle files created with protocols >= 1 in
binary mode. For the old ASCII-based pickle protocol 0 you can use
either text mode or binary mode as long as you stay consistent.

A pickle file written with protocol 0 in binary mode will contain lone
linefeeds as line terminators and therefore will look “funny” when
viewed in Notepad or other editors which do not support this format."

Date User Action Args
2008-09-18 21:04:49benjamin.petersonsetstatus: open -> closed
resolution: not a bug
messages: + msg73400
nosy: + benjamin.peterson
2008-09-18 20:58:34Georgcreate