Author vstinner
Recipients BreamoreBoy, alexandre.vassalotti, amaury.forgeotdarc, collinwinter, hagen, pitrou, vstinner
Date 2010-07-28.22:58:01
SpamBayes Score 0.000132123
Marked as misclassified No
Message-id <1280357884.17.0.668768771209.issue3873@psf.upfronthosting.co.za>
In-reply-to
Content
New version of my patch:
 - add "used" attribute to UnpicklerBuffer structure: disable the read buffer for not seekable file and for protocol 0 (at the first call to unpickle_readline)
 - check if PyObject_GetAttrString(file, "seek") is NULL or not
 - unpickle_readline() flushs also the buffer
 - add a new patch specific to the read buffer: ensure that unpickler doesn't eat data at the end of the file

test_pickle pass without any error.

Disable read buffer at the first call to unpickle_readline() because unpickle_readline() have to flush the buffer. I will be very difficult to optimize protocol 0, but I hope that nobody uses it nowadays.

===========

Benchmark with [0]*10**6, Python compiled with pydebug.

Without the patch
-----------------

Protocol 0:
- dump: 598.0 ms
- load (seekable=False): 3337.3 ms
- load (seekable=True): 3309.6 ms

Protocol 1:
- dump: 217.8 ms
- load (seekable=False): 864.2 ms
- load (seekable=True): 873.3 ms

Protocol 2:
- dump: 226.5 ms
- load (seekable=False): 867.8 ms
- load (seekable=True): 854.6 ms


With the patch
--------------

Protocol 0
- dump: 615.5 ms
- load (seekable=False): 3201.3 ms
- load (seekable=True): 3223.4 ms

Protocol 1
- dump: 219.8 ms
- load (seekable=False): 942.1 ms
- load (seekable=True): 175.2 ms

Protocol 2
- dump: 221.1 ms
- load (seekable=False): 943.9 ms
- load (seekable=True): 175.5 ms
History
Date User Action Args
2010-07-28 22:58:04vstinnersetrecipients: + vstinner, collinwinter, amaury.forgeotdarc, pitrou, alexandre.vassalotti, hagen, BreamoreBoy
2010-07-28 22:58:04vstinnersetmessageid: <1280357884.17.0.668768771209.issue3873@psf.upfronthosting.co.za>
2010-07-28 22:58:02vstinnerlinkissue3873 messages
2010-07-28 22:58:01vstinnercreate