classification
Title: Implementing Async IO
Type: enhancement Stage: needs patch
Components: Extension Modules Versions: Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: jcea Nosy List: brian.curtin, dabeaz, giampaolo.rodola, jcea, pitrou, yselivanov
Priority: normal Keywords:

Created on 2011-02-04 15:26 by jcea, last changed 2014-01-31 21:47 by yselivanov. This issue is now closed.

Messages (6)
msg127898 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-02-04 15:26
This issue is a placeholder for exposing Async IO thru Python.

The relevant mail thread is at:

http://mail.python.org/pipermail/python-dev/2010-October/104770.html


Relevant resources:

http://feedproxy.google.com/~r/Wwwc0t0d0s0org/~3/snM0BJ8evv4/7147-Synchronicity-in-asynchronicity.html

http://www.opengroup.org/onlinepubs/000095399/basedefs/aio.h.html
msg127899 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-04 15:41
From the python-dev thread:

http://mail.python.org/pipermail/python-dev/2010-October/104782.html

“os._exit is useful. os.open is useful. aio_* are *not* useful. For anything. If there's anything you think you want to use them for, you're wrong. It either won't work properly or it will worse performing than the simpler alternatives.”

http://mail.python.org/pipermail/python-dev/2010-October/104786.html

“I'd like to echo this sentiment.  This is not about providing a 'safe' wrapper to hide some powerful feature of these APIs: the POSIX aio_* functions are really completely useless.

[...]

Not only is the performance usually worse than expected, the behavior of aio_* functions require all kinds of subtle and mysterious coordination with signal handling, which I'm not entirely sure Python would even be able to pull off without some modifications to the signal module.  (And, as Jean-Paul mentioned, if your OS kernel runs out of space in a queue somewhere, completion notifications might just never be delivered at all.)”
msg127987 - (view) Author: David Beazley (dabeaz) Date: 2011-02-05 13:42
Anyone contemplating the use of aio_ functions should first go read "The Story of Mel".

http://www.catb.org/jargon/html/story-of-mel.html
msg127990 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-02-05 14:11
Thanks, David, for the link. The story is really touching :-).
msg127994 - (view) Author: David Beazley (dabeaz) Date: 2011-02-05 14:54
Glad you liked it!   I think there is a bit of a cautionary tale in there though. With aio_, there is the promise of better performance, but you're also going to need a *LOT* of advance planning and thought to avoid creating a tangled coding nightmare with it.

Just as an aside, one of the uses of aio_ related functions is to implement parts of user-level thread libraries in C (e.g., pthreads, etc.). A library might use the asynchronous I/O callbacks as part of implementing non-kernel (green) threads.  The code for doing this tends to be very low level and hairy with lots of signal handling--for example, if you want to context-switch between two user-level threads in C, you usually do it inside a signal handler (i.e., you thread-switch inside the signal handler called in response to aio_ completions). 

Whether it's feasible to expose aio_* all the way up to Python or not is an open question. I suspect it will be fraught with lots of tricky issues. In the end, it might just be easier to use threads.  Nevertheless, you'll learn a lot about Python internals by working on this :-).
msg209833 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2014-01-31 21:47
I guess PEP 3156 covers this, closing this one.
History
Date User Action Args
2014-01-31 21:47:18yselivanovsetstatus: open -> closed

nosy: + yselivanov
messages: + msg209833

resolution: out of date
2011-02-05 14:54:41dabeazsetnosy: jcea, pitrou, giampaolo.rodola, brian.curtin, dabeaz
messages: + msg127994
2011-02-05 14:11:56jceasetnosy: jcea, pitrou, giampaolo.rodola, brian.curtin, dabeaz
messages: + msg127990
2011-02-05 13:42:05dabeazsetnosy: + dabeaz
messages: + msg127987
2011-02-04 16:31:11giampaolo.rodolasetnosy: + giampaolo.rodola
2011-02-04 15:41:31pitrousetnosy: + pitrou
messages: + msg127899
2011-02-04 15:37:32brian.curtinsetnosy: + brian.curtin
2011-02-04 15:26:37jceacreate