Author neologix
Recipients Albert.Strasheim, aljungberg, asksol, bquinlan, brian.curtin, gdb, gkcn, haypo, hongqn, jcea, jnoller, neologix, pitrou, vlasovskikh
Date 2011-05-14.11:50:33
SpamBayes Score 4.01852e-06
Marked as misclassified No
Message-id <>
In-reply-to <1305292501.3561.24.camel@localhost.localdomain>
> Indeed, it isn't, Pipe objects are not meant to be safe against multiple
> access. Queue objects (in multiprocessing/ use locks so they
> are safe.

But if the write to the Pipe is not atomic, then the select isn't safe.
select will return as soon as some data is available for reading. So
let's say this happens:
parent process waiting inside Connection.recv() on poll():
def recv(self, sentinels=None):
        if sentinels:
            self._poll(-1.0, sentinels)
        buf = self._recv_bytes()
        return pickle.loads(buf.getbuffer())

worker process, inside Connection.send():
def send_bytes(self, buf):
        n = len(buf)
        self._send(struct.pack("=i", len(buf)))
        # The condition is necessary to avoid "broken pipe" errors
        # when sending a 0-length buffer if the other end closed the pipe.
        if n > 0:

In the parent process, _poll() will return self._handle as readable as
soon as len(buf) has been sent by the worker process. Thus,
Connection.recv_bytes() will be called:

    def _recv_bytes(self, maxsize=None):
        buf = self._recv(4)
        size, = struct.unpack("=i", buf.getvalue())
        if maxsize is not None and size > maxsize:
            return None
        return self._recv(size)

The first _recv() will succeed, since the length is in the Pipe.
The second one, however, will remain stuck on the read from the
pipe/unix socket, because there's no more data.
This can be reproduced easily by adding a short sleep right after the
sending of the length of the buffer inside send_bytes(), and then
sending a SIGKILL to a worker process.
(Also, I now remember why I made the comment about the handle being
read after the select, I spotted this earlier but completely forgot
about it afterwards...).
Date User Action Args
2011-05-14 11:50:34neologixsetrecipients: + neologix, jcea, bquinlan, pitrou, haypo, jnoller, hongqn, brian.curtin, asksol, vlasovskikh, gdb, Albert.Strasheim, aljungberg, gkcn
2011-05-14 11:50:33neologixlinkissue9205 messages
2011-05-14 11:50:33neologixcreate