This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: pickle.Unpickler may read too many bytes, causing hangs with blocking input stream
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.3, Python 3.4
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: alexandre.vassalotti Nosy List: alexandre.vassalotti, dylanr, jm, pitrou, serhiy.storchaka
Priority: normal Keywords:

Created on 2013-05-27 18:04 by jm, last changed 2022-04-11 14:57 by admin. This issue is now closed.

File name Uploaded Description Edit jm, 2013-05-27 18:04
Messages (5)
msg190149 - (view) Author: Julien Muchembled (jm) Date: 2013-05-27 18:04
I use pickle over sockets to make 2 processes communicate and I experience hangs when the resulting pickle message is exactly 4097 bytes.

Attached file is a small script reproducing the issue, trying a few values around 4096. It runs on both Python 2.7 & 3.3:

- Python 2.7.3: ok (script exits)
- Python 3.3.2: last iteration hangs

With strace, you can easily see that after reading the 4097 bytes:

[pid 17100] read(3,  <unfinished ...>
[pid 17100] <... read resumed> "\200\2X\367\17\0\0                         "..., 4096) = 4096
[pid 17100] read(3, ".", 4096)          = 1

it reads again without reason:

[pid 17100] read(3, 

No issue with the Python implementation, that you can try with little change:

- Python 2.7.3: ok   (from pickle import Unpickler)
- Python 3.3.2: ok   (from pickle import _Unpickler as Unpickler)
msg192628 - (view) Author: Dylan Radcliffe (dylanr) Date: 2013-07-08 08:15
What I see in gdb is that that read is occurring due to the unpickling thread reading peeking ahead in the stream due to buffered io being used.

#0  0x00007ffff7bcbcbd in read () from /lib/x86_64-linux-gnu/
#1  0x000000000061f521 in fileio_readinto (self=0x7ffff6491528, args=0x7ffff66dc190) at ./Modules/_io/fileio.c:541
#2  0x00000000004b96f0 in PyCFunction_Call (func=0x7ffff601d598, arg=0x7ffff66dc190, kw=0x0) at Objects/methodobject.c:90
#3  0x000000000045a0c2 in PyObject_Call (func=0x7ffff601d598, arg=0x7ffff66dc190, kw=0x0) at Objects/abstract.c:2105
#4  0x000000000045ad3e in PyObject_CallMethodObjArgs (callable=0x7ffff601d598, name=0x7ffff7f3ccf0) at Objects/abstract.c:2341
#5  0x0000000000626e65 in _bufferedreader_raw_read (self=0x7ffff6777e58, start=0xa89210 ".\002X\367\017", len=4096) at ./Modules/_io/bufferedio.c:1438
#6  0x00000000006270ed in _bufferedreader_fill_buffer (self=0x7ffff6777e59) at ./Modules/_io/bufferedio.c:1470
#7  0x0000000000628474 in _bufferedreader_peek_unlocked (self=0x7ffff6777e58) at ./Modules/_io/bufferedio.c:1714
#8  0x0000000000624b1d in buffered_peek (self=0x7ffff6777e58, args=0x7ffff66d4948) at ./Modules/_io/bufferedio.c:880

As a possible workaround if you disable buffering when opening your pipe for input it seems to prevent the problem (though I am not sure about performance implications).

< load = Unpickler(os.fdopen(r, 'rb')).load
> load = Unpickler(os.fdopen(r, 'rb', 0)).load
msg192629 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-07-08 08:24
For the record, this is one of the things that the framing proposal in pickle protocol 4 (pep 3154 - is able to address.
msg193210 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-07-17 08:47
Seems as the patch for issue17897 fixes this case.
msg205007 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-02 12:32
If issue17897 fixes this bug (I'm not absolutely sure), perhaps it should be backported to 3.3.
Date User Action Args
2022-04-11 14:57:46adminsetgithub: 62273
2013-12-02 12:32:49serhiy.storchakasetmessages: + msg205007
2013-12-02 01:12:58alexandre.vassalottisetstatus: open -> closed
assignee: alexandre.vassalotti
resolution: fixed
stage: resolved
2013-07-17 08:47:41serhiy.storchakasetmessages: + msg193210
2013-07-08 08:24:28pitrousetmessages: + msg192629
2013-07-08 08:15:23dylanrsetnosy: + dylanr
messages: + msg192628
2013-05-27 18:34:00serhiy.storchakasetnosy: + pitrou, alexandre.vassalotti, serhiy.storchaka

versions: + Python 3.4
2013-05-27 18:04:41jmcreate