classification
Title: Add I/O Completion Ports wrapper
Type: enhancement Stage: resolved
Components: Extension Modules, Windows Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: brian.curtin, christian.heimes, exarkun, giampaolo.rodola, gvanrossum, jcea, jkloth, mhammond, pitrou, sbt
Priority: normal Keywords: patch

Created on 2012-10-09 16:09 by christian.heimes, last changed 2013-10-21 13:00 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
iocp.patch sbt, 2012-10-10 13:13 review
iocp_example.py sbt, 2012-10-10 13:14
Messages (18)
msg172489 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-09 16:09
Guido has expressed [1] that he like to see IOCP support in the stdlib for the grant unified asyncore interface.

Quote from MSDN [2]:
---
I/O completion ports provide an efficient threading model for processing multiple asynchronous I/O requests on a multiprocessor system. When a process creates an I/O completion port, the system creates an associated queue object for requests whose sole purpose is to service these requests. Processes that handle many concurrent asynchronous I/O requests can do so more quickly and efficiently by using I/O completion ports in conjunction with a pre-allocated a  thread pool than by creating threads at the time they receive an I/O request.
---

I've found a couple of interfaces to IOCP in Python projects. Twisted [3] has a Cython based interface to the IOCP API, Cogen [4] uses ctypes wrapper. I couldn't find IOCP support in Tornado. I favor a C implementation over ctypes. What's our attitude to Cython? Personally I like Cython and use it in several projects but I'm not sure if we shall use it in core development.

Christian

[1] http://mail.python.org/pipermail/python-ideas/2012-October/016539.html
[2] http://msdn.microsoft.com/en-us/library/aa365198%28VS.85%29.aspx
[3] http://twistedmatrix.com/trac/browser/trunk/twisted/internet/iocpreactor/iocpsupport/
[4] http://code.google.com/p/cogen/source/browse/trunk/cogen/core/proactors
msg172492 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-10-09 16:37
> Cogen [4] uses ctypes wrapper.

In the code for the IOCP reactor only ctypes.FormatError() is used from ctypes.  It uses pywin32 instead.
msg172494 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2012-10-09 16:42
Tough choice.  I'm not in favor of using either ctypes or Cython for this purpose -- ctypes because it's brittle, and Cython because it is a huge complicated system of its own that I would rather not depend on.  Cython already depends on CPython, so CPython depending on Cython would essentially marry the two systems.

How hard could it be to write a C++ extension wrapping IOCP?  From reading the docs there are only a handful API methods (of which the main one stands out as a nadir of API design -- it's like the designers were told they could only add one function... :-).

However a bigger problem probably is that it only makes sense if you also wrap the rest of the handle-based I/O functionality on Windows.  The docs talk about "overlapping" I/O which I presume is a form of async I/O.  Most likely we'll have to look at Mark Hammond's venerable win32 package for that.  Maybe it makes most sense to have IOCP integrated there?  (For all I know it's already supported...)

The main think I want to be sure of is to design the abstract I/O loop (aka reactor) general enough that it will be easy to hook in IOCP-based event-generating and -handling components.
msg172495 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-09 16:58
Guido,

Richard pointed out that pywin32 already wraps the necessary bits of IOCP. The functions are in the win32file package. Cogen's reactor uses pywin32 instead of ctypes, too. The library has an *additional* ctypes based interface in the directory ctypes_iocp_impl.
msg172496 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2012-10-09 17:07
So do we need this ticket at all?  It seems there's no code to write -- all we need to do is make sure we can integrate IOCP in the future standard reactor interface.  That hardly seems a reason to keep a ticket open.
msg172497 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-10-09 17:15
Note that since Python 3.3, multiprocessing and _winapi make some use of overlapped IO.

One can use _winapi.ReadFile() and _winapi.WriteFile() to do overlapped IO on normal socket handles created using socket.socket().
msg172502 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-09 18:41
I'm not sure why we would have "no code to write" unless you're thinking of integrating pywin32 into the stdlib.
As Richard said, overlapped I/O is already more or less supported as part of the _winapi private module. But IOCP itself isn't exposed. We could have a public module exposing IOCP as a nice API :-)
msg172519 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2012-10-09 20:27
> We could have a public module exposing IOCP as a nice API :-)

What about select module?
msg172523 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2012-10-09 20:45
I would be happy with requiring that the user use pywin32 if they want to use this.

Have you all read the docs for IOCP?  It is not for the faint of heart.  E.g. it integrates with the thread scheduler.
msg172524 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-09 20:59
> Have you all read the docs for IOCP?  It is not for the faint of
> heart.  E.g. it integrates with the thread scheduler.

Yes, IOCP (or overlapped I/O) may use threads under the hood, but IIUC
this can be ignored by the programmer, and GetQueuedCompletionStatus()
acts as the select()-like function (this is assuming you create a single
I/O cooperation port and register all your socket handles to this single
object).
msg172526 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2012-10-09 21:09
According to http://msdn.microsoft.com/en-us/library/aa365198%28VS.85%29.aspx once you call GetQueuedCompletionStatus() your thread is managed by the IOCP and only up to a given parameter of those threads are allowed to run.  They recommend setting that parameter to the number of CPUs.  This feels scary (doubly so in the light of the GIL).
msg172528 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-09 21:16
> According to http://msdn.microsoft.com/en-us/library/aa365198%28VS.85%
> 29.aspx once you call GetQueuedCompletionStatus() your thread is
> managed by the IOCP and only up to a given parameter of those threads
> are allowed to run.  They recommend setting that parameter to the
> number of CPUs.  This feels scary (doubly so in the light of the GIL).

I think this mostly means that you should always call
GetQueuedCompletionStatus() from the same thread for a given IO
completion port. Which, in the context of a single-threaded event loop,
shouldn't be a problem (also, this can be enforced by our stdlib
wrapper).

AFAIU, the MSDN docs have this complicated language about threads mostly
because they suggest you to use as many threads as there are CPUs on the
machines (in order to max out the I/O processing bandwidth).

By the way, I've just checked: the Twisted IOCP reactor uses a single
I/O completion port to which it registers all socket handles, and its
event loop calls GetQueuedCompletionStatus() in a loop.
msg172539 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-10-09 22:12
If this is going to be available for Windows, I would like to help to integrate too the "Event Completion Framework" of Solaris 10 and up.
msg172540 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-10-09 22:13
Some old documentation: http://web.archive.org/web/20110719052845/http://developers.sun.com/solaris/articles/event_completion.html
msg172571 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-10-10 13:13
Adding the IOCP functions to _winapi is straightforward -- see patch.  Note that there seems to be no way to unregister a handle from an IOCP.

Creating overlapped equivalents of socket.accept() and socket.connect() looks more complicated.  Perhaps that should be added to the socket module or left to pywin32.
msg200766 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 12:28
Is this patch still of relevance for asyncio?
msg200769 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-10-21 12:47
> Is this patch still of relevance for asyncio?

No, the _overlapped extension contains the IOCP stuff.
msg200773 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 13:00
Thanks, I'm closing this ticket.
History
Date User Action Args
2013-10-21 13:00:19christian.heimessetstatus: open -> closed
resolution: out of date
messages: + msg200773

stage: resolved
2013-10-21 12:47:18sbtsetmessages: + msg200769
2013-10-21 12:28:32christian.heimessetmessages: + msg200766
2012-10-10 13:14:48sbtsetfiles: + iocp_example.py
2012-10-10 13:13:35sbtsetfiles: + iocp.patch
keywords: + patch
messages: + msg172571
2012-10-09 22:13:52jceasetmessages: + msg172540
2012-10-09 22:12:21jceasetmessages: + msg172539
2012-10-09 22:11:16jceasetnosy: + jcea
2012-10-09 21:16:12pitrousetmessages: + msg172528
2012-10-09 21:09:40gvanrossumsetmessages: + msg172526
2012-10-09 20:59:26pitrousetmessages: + msg172524
2012-10-09 20:45:51gvanrossumsetmessages: + msg172523
2012-10-09 20:27:09giampaolo.rodolasetnosy: + giampaolo.rodola
messages: + msg172519
2012-10-09 18:41:53pitrousetnosy: + pitrou
messages: + msg172502
2012-10-09 17:45:21brian.curtinsetnosy: + brian.curtin
2012-10-09 17:15:53sbtsetmessages: + msg172497
2012-10-09 17:07:10gvanrossumsetmessages: + msg172496
2012-10-09 16:58:35christian.heimessetmessages: + msg172495
2012-10-09 16:53:35jklothsetnosy: + jkloth
2012-10-09 16:43:24gvanrossumsetnosy: + mhammond
2012-10-09 16:42:59gvanrossumsetmessages: + msg172494
2012-10-09 16:37:26sbtsetnosy: + sbt
messages: + msg172492
2012-10-09 16:09:14christian.heimescreate