Author njs
Recipients njs
Date 2008-12-23.12:30:06
SpamBayes Score 1.32907e-08
Marked as misclassified No
Message-id <>
cPickle.dump by default does not properly encode unicode characters
outside the BMP -- it throws away the high bits:

>>> cPickle.loads(cPickle.dumps(u"\U00012345"))

The problem is in dump, not load:
>>> pickle.dumps(u"\U00012345")   # works
>>> cPickle.dumps(u"\U00012345")  # no!

It does work correctly when using a more modern pickling protocol:

>>> cPickle.loads(cPickle.dumps(u"\U00012345", 1))
>>> cPickle.loads(cPickle.dumps(u"\U00012345", 2))

But this is not much comfort for users whose data has been corrupted
because they went with the defaults.  (Fortunately in my application I
knew that all my characters were in the supplementary plane, so I could
repair the data after the fact, but...)

Above tests are with 2.5.2, but from checking the source, the bug is
obviously still present in 2.6.1:
cPickle.c:modified_EncodeRawUnicodeEscape has no code to handle 32-bit
unicode values.

OTOH, it does look like someone noticed the problem and fixed it for
3.0; _pickle.c:raw_unicode_escape handles such characters fine.  Guess
they just forgot to backport the fixes... but the code is there, and can
probably just be copy-pasted back to 2.6.
Date User Action Args
2008-12-23 12:30:46njssetrecipients: + njs
2008-12-23 12:30:46njssetmessageid: <>
2008-12-23 12:30:08njslinkissue4730 messages
2008-12-23 12:30:06njscreate