Author vstinner
Recipients lemburg, loewis, vstinner
Date 2010-04-13.00:39:53
SpamBayes Score 1.897e-11
Marked as misclassified No
Message-id <>
Python3 uses unicode surrogates to store undecodable filenames. Eg. the filename b"abc\" is encoded as "abc\" if the file system encoding is ASCII. Pickle is unable to store them:

./python -c 'import pickle; pickle.dumps("abc\udcff")'
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 20: surrogates not allowed

This is a limitation of pickle (in the binary mode): Python accepts to store any unicode character, but pickle doesn't.

Using "surrogatepass" error handler should be enough to fix this issue.

Related issue: #3672 (Reject surrogates in utf-8 codec) -> r72208 creates "surrogatepass" error handler.
Date User Action Args
2010-04-13 00:39:57vstinnersetrecipients: + vstinner, lemburg, loewis
2010-04-13 00:39:57vstinnersetmessageid: <>
2010-04-13 00:39:55vstinnerlinkissue8383 messages
2010-04-13 00:39:54vstinnercreate