Message111867
New version of my patch:
- add "used" attribute to UnpicklerBuffer structure: disable the read buffer for not seekable file and for protocol 0 (at the first call to unpickle_readline)
- check if PyObject_GetAttrString(file, "seek") is NULL or not
- unpickle_readline() flushs also the buffer
- add a new patch specific to the read buffer: ensure that unpickler doesn't eat data at the end of the file
test_pickle pass without any error.
Disable read buffer at the first call to unpickle_readline() because unpickle_readline() have to flush the buffer. I will be very difficult to optimize protocol 0, but I hope that nobody uses it nowadays.
===========
Benchmark with [0]*10**6, Python compiled with pydebug.
Without the patch
-----------------
Protocol 0:
- dump: 598.0 ms
- load (seekable=False): 3337.3 ms
- load (seekable=True): 3309.6 ms
Protocol 1:
- dump: 217.8 ms
- load (seekable=False): 864.2 ms
- load (seekable=True): 873.3 ms
Protocol 2:
- dump: 226.5 ms
- load (seekable=False): 867.8 ms
- load (seekable=True): 854.6 ms
With the patch
--------------
Protocol 0
- dump: 615.5 ms
- load (seekable=False): 3201.3 ms
- load (seekable=True): 3223.4 ms
Protocol 1
- dump: 219.8 ms
- load (seekable=False): 942.1 ms
- load (seekable=True): 175.2 ms
Protocol 2
- dump: 221.1 ms
- load (seekable=False): 943.9 ms
- load (seekable=True): 175.5 ms |
|
Date |
User |
Action |
Args |
2010-07-28 22:58:04 | vstinner | set | recipients:
+ vstinner, collinwinter, amaury.forgeotdarc, pitrou, alexandre.vassalotti, hagen, BreamoreBoy |
2010-07-28 22:58:04 | vstinner | set | messageid: <1280357884.17.0.668768771209.issue3873@psf.upfronthosting.co.za> |
2010-07-28 22:58:02 | vstinner | link | issue3873 messages |
2010-07-28 22:58:01 | vstinner | create | |
|