classification
Title: Optimize unpickle prefetching
Type: performance Stage: resolved
Components: Extension Modules Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: alexandre.vassalotti, haypo, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-05-03 18:16 by serhiy.storchaka, last changed 2013-11-30 21:17 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
pickle_peek.patch serhiy.storchaka, 2013-05-03 18:16 review
pickle_peek_2.patch serhiy.storchaka, 2013-11-30 18:09 review
Messages (11)
msg188317 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-03 18:16
When C implementation of the unpickle reads a data, it also prefetches some data using peek() (if available) and then concatenates read and peeked chunks in the one input buffer. This causes an additional copying when a large data is read. The proposed patch gets rid of concatenating by moving a peeking before reading.
msg188335 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-05-03 23:50
Do you have benchmark results to show the code with the patch is faster?
msg188343 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-04 08:08
No, I have no any interesting results. ;) If an effect exists, it should reveal itself only in rare cases and be very small. This patch may be considered rather as a tiny refactoring (it decreases a number of code lines).
msg193212 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-07-17 08:48
See issue18073 for test case.
msg193214 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-07-17 08:52
I'd prefer to see efforts go towards finishing the PEP 3154 implementation...
msg193215 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-07-17 09:04
This patch is orthogonal to the PEP 3154 framing. The affecting of issue18073 is only an unexpected side effect.
msg204832 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-30 18:09
Patch is synchronized with tip (it was desynchronized since 23459df0753e).
msg204833 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-30 18:32
Microbenchmark:

$ ./python -c "import pickle; d = b'x' * 10**6; f = open('test.pickle3', 'wb'); pickle.dump(d, f, 3); f.close()"
$ ./python -m timeit -s "from pickle import load"  "with open('test.pickle3', 'rb') as f: load(f)"

Unpatched:  100 loops, best of 3: 7.27 msec per loop
Patched:    100 loops, best of 3: 4.87 msec per loop
msg204843 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-11-30 20:53
Looks good to me! Feel free to commit.
msg204846 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-30 21:16
New changeset d565310e7ae3 by Serhiy Storchaka in branch 'default':
Issue #17897: Optimized unpickle prefetching.
http://hg.python.org/cpython/rev/d565310e7ae3
msg204847 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-30 21:17
Thank you Alexandre.
History
Date User Action Args
2013-11-30 21:17:26serhiy.storchakasetassignee: serhiy.storchaka
2013-11-30 21:17:14serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg204847

stage: patch review -> resolved
2013-11-30 21:16:12python-devsetnosy: + python-dev
messages: + msg204846
2013-11-30 20:53:27alexandre.vassalottisetmessages: + msg204843
2013-11-30 18:32:18serhiy.storchakasetmessages: + msg204833
2013-11-30 18:09:28serhiy.storchakasetfiles: + pickle_peek_2.patch

messages: + msg204832
2013-07-17 09:04:49serhiy.storchakasetmessages: + msg193215
2013-07-17 08:52:29pitrousetnosy: - pitrou
2013-07-17 08:52:20pitrousetmessages: + msg193214
2013-07-17 08:48:28serhiy.storchakasetmessages: + msg193212
2013-05-04 08:08:09serhiy.storchakasetmessages: + msg188343
2013-05-04 00:03:24hayposetnosy: + haypo
2013-05-03 23:50:37alexandre.vassalottisetmessages: + msg188335
2013-05-03 18:16:39serhiy.storchakacreate