Author pitrou
Recipients alexandre.vassalotti, irmen, pitrou
Date 2011-11-30.02:21:46
SpamBayes Score 8.29569e-09
Marked as misclassified No
Message-id <1322619707.99.0.479265394652.issue13505@psf.upfronthosting.co.za>
In-reply-to
Content
In Python 3.2:

>>> pickle.dumps(b'xyz', protocol=2)
b'\x80\x02c__builtin__\nbytes\nq\x00]q\x01(KxKyKze\x85q\x02Rq\x03.'

In Python 2.7:

>>> pickle.loads(b'\x80\x02c__builtin__\nbytes\nq\x00]q\x01(KxKyKze\x85q\x02Rq\x03.')
'[120, 121, 122]'

The problem is that the bytes() constructor argument is a list of ints, which gives a different result when reconstructed under 2.x where bytes is an alias of str:

>>> pickletools.dis(pickle.dumps(b'xyz', protocol=2))
    0: \x80 PROTO      2
    2: c    GLOBAL     '__builtin__ bytes'
   21: q    BINPUT     0
   23: ]    EMPTY_LIST
   24: q    BINPUT     1
   26: (    MARK
   27: K        BININT1    120
   29: K        BININT1    121
   31: K        BININT1    122
   33: e        APPENDS    (MARK at 26)
   34: \x85 TUPLE1
   35: q    BINPUT     2
   37: R    REDUCE
   38: q    BINPUT     3
   40: .    STOP
highest protocol among opcodes = 2

Bytearray objects use a different trick: they pass a (unicode string, encoding) pair which has the same constructor semantics under 2.x and 3.x. Additionally, such encoding is statistically more efficient: a list of 1-byte ints will take 2 bytes per encoded char, while a latin1-to-utf8 transcoded string (BINUNICODE uses utf-8) will take on average 1.5 bytes per encoded char (assuming a 50% probability of higher-than-127 bytes).

>>> pickletools.dis(pickle.dumps(bytearray(b'xyz'), protocol=2))
    0: \x80 PROTO      2
    2: c    GLOBAL     '__builtin__ bytearray'
   25: q    BINPUT     0
   27: X    BINUNICODE 'xyz'
   35: q    BINPUT     1
   37: X    BINUNICODE 'latin-1'
   49: q    BINPUT     2
   51: \x86 TUPLE2
   52: q    BINPUT     3
   54: R    REDUCE
   55: q    BINPUT     4
   57: .    STOP
highest protocol among opcodes = 2
History
Date User Action Args
2011-11-30 02:21:48pitrousetrecipients: + pitrou, irmen, alexandre.vassalotti
2011-11-30 02:21:47pitrousetmessageid: <1322619707.99.0.479265394652.issue13505@psf.upfronthosting.co.za>
2011-11-30 02:21:47pitroulinkissue13505 messages
2011-11-30 02:21:46pitroucreate