classification
Title: _multiprocessing.Connection.poll with timeout uses polling under Windows
Type: resource usage Stage: resolved
Components: Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder: Parent process hanging in multiprocessing if children terminate unexpectedly
View: 9205
Assigned To: brian.curtin Nosy List: asksol, brian.curtin, jnoller, kristjan.jonsson, pitrou, tim.golden
Priority: normal Keywords: patch

Created on 2011-03-25 00:27 by pitrou, last changed 2012-04-01 14:03 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
issue11668.diff brian.curtin, 2011-03-26 04:09 review
Messages (10)
msg132058 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-25 00:27
This impacts higher-level APIs such as Queue.get() with a positive timeout.
I'm not sure whether it's easily possible to fix this. A Google search seems to suggest that the pipe could be opened with "SYNCHRONIZE" access rights (?) and then WaitForSingleObject() be used on the handle.
msg132059 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-25 00:32
For the record, the polling code is in conn_poll() in Modules/_multiprocessing/pipe_connection.c.
Windows named pipes seem to be created in pure Python in Lib/multiprocessing/connection.py.
msg132069 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-03-25 03:55
SYNCHRONIZE comes for free on pipes created with CreateNamedPipe, so there's nothing to do there.

I think it's more likely that we'll have to use WaitForMultipleObjects and include the pipe handle with a signal handler for Ctrl-C. I believe this is done elsewhere in the code, timemodule comes to mind (for sleep).

I'll see what I can come up with.
msg132215 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-03-26 04:09
Attaching an initial patch implementing the same functionality but using WaitForMultipleObjects. Here's some details:

WFMO takes an array of objects to wait on, which is the pipe handle and sigint_event which is a handle to an event which gets set when CTRL-C is caught (see Modules/_multiprocessing/multiprocessing.c). Waiting for both objects replaces the need to write a custom loop which periodically calls PyErr_CheckSignals.

A negative timeout was effectively a blocking call, so we can let WFMO handle that with an INFINITE timeout setting.

WFMO returns the object which raised it relative from event 0, so the switch case for a CTRL-C is 0+1 and returns the same value as before. If the pipe was the object to raise, just like before: return true if there's data, false if not.
msg132217 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2011-03-26 08:18
I don't think it is necessary to have the initial call to PeekNamedPipe.  It is a kernel call just like WFMO and so just as expensive.  It also has some strange semantics, (such as being possibly blocking in a multithreaded application.  read the docs...).  Just go for the WFMO to keep things simple.
msg132238 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-26 13:49
Are you sure about MP_EXCEPTION_HAS_BEEN_SET?
semaphore.c has a more sophisticated logic:

        Py_BEGIN_ALLOW_THREADS
        ResetEvent(sigint_event);
        res = WaitForMultipleObjects(2, handles, FALSE, msecs);
        Py_END_ALLOW_THREADS

        /* handle result */
        if (res != WAIT_OBJECT_0 + 1)
            break;

        /* got SIGINT so give signal handler a chance to run */
        Sleep(1);

        /* if this is main thread let KeyboardInterrupt be raised */
        if (PyErr_CheckSignals())
            return NULL;

        /* recalculate timeout */
        if (msecs != INFINITE) {
            ticks = GetTickCount();
            if ((DWORD)(ticks - start) >= full_msecs)
                Py_RETURN_FALSE;
            msecs = full_msecs - (ticks - start);
        }
msg135393 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-07 00:39
This doesn't seem to be so easy. WFMO (or WFSO) often seems to return successfully while there's no message to read on the pipe end. Here is a sample session (disturbing results):

>>> a, b = connection.Pipe(True)
>>> win32.WaitForMultipleObjects([a._handle], True, 1000)
258
>>> win32.WaitForMultipleObjects([b._handle], True, 1000)
0
>>> win32.PeekNamedPipe(a._handle)
(0, 0)
>>> win32.WaitForMultipleObjects([a._handle], True, 1000)
0

(do note how the end created with CreateFile() is considered ready by WaitForMultipleObjects, while the end created with CreateNamedPipe() is considered ready after an unsuccessful call to PeekNamedPipe()!)
msg135434 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-07 10:00
A solution could be to use overlapped I/O on the named pipe handles.
The first poll() would call ReadFile() with a tiny value (1?) and store an object wrapping the OVERLAPPED structure. Subsequent poll() or recv() would re-use that structure until the overlapped read succeeds.
The hEvent in the OVERLAPPED structure should be usable in WFMO fine.
msg135494 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-07 17:52
I have a full patch using overlapped I/O in issue9205 (sentinels3.patch).
msg157289 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-04-01 14:03
This is totally outdated by the new Connection implementation in 3.3.
History
Date User Action Args
2012-04-01 14:03:25pitrousetstatus: open -> closed
superseder: Parent process hanging in multiprocessing if children terminate unexpectedly
messages: + msg157289

resolution: out of date
stage: resolved
2011-05-07 17:52:21pitrousetmessages: + msg135494
2011-05-07 10:00:03pitrousetmessages: + msg135434
2011-05-07 00:39:43pitrousetmessages: + msg135393
2011-03-26 13:49:23pitrousetmessages: + msg132238
2011-03-26 08:18:53kristjan.jonssonsetmessages: + msg132217
2011-03-26 04:09:59brian.curtinsetfiles: + issue11668.diff
keywords: + patch
messages: + msg132215
2011-03-25 03:55:24brian.curtinsetassignee: brian.curtin
messages: + msg132069
2011-03-25 00:32:48pitrousetmessages: + msg132059
2011-03-25 00:27:34pitroucreate