classification
Title: Use the new selectors module in the subprocess module
Type: enhancement Stage: resolved
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: 19172 Superseder:
Assigned To: Nosy List: giampaolo.rodola, neologix, pitrou, python-dev, vstinner
Priority: normal Keywords: needs review, patch

Created on 2013-09-04 18:41 by vstinner, last changed 2013-11-09 06:48 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess_selectors.diff neologix, 2013-10-05 14:42 review
subprocess_selectors-1.diff neologix, 2013-10-30 19:39 review
subprocess_selectors-2.diff neologix, 2013-11-01 10:37 review
subprocess_selectors-3.diff neologix, 2013-11-05 19:40 review
Messages (8)
msg196936 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-09-04 18:41
Python 3.4 has a new selectors module (issue #16853). It would be nice to use it instead of select.poll or select.select in the subprocess module.
msg196937 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-09-04 18:44
Other modules using select.select() or select.poll() for more than 1 file descriptor:

- asyncore
- multiprocessing.connection
- multiprocessing.forkserver
msg196995 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2013-09-05 13:08
It's likely that asyncore won't be able to take any practical advantage from this integration.

To say one, epoll()/kqueue() pollers won't bring any speedup over select()/poll() because of how asyncore.loop() function is implemented (see http://bugs.python.org/issue6692#msg103628 and http://bugs.python.org/issue11273).

Also, the new selectors module only takes read and write events into account, whereas asyncore explicitly closes dispatcher in case of disconnection events (POLLOUT, etc).

In summary I'd say it's a lot wiser to leave asyncore alone and consider it frozen.
msg198989 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-10-05 14:42
Here's a patch updating subprocess to use selectors.
(It depends on the new keys() method - issue #19172.)
msg201774 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-10-30 19:39
Here's an updated patch using the new selector.get_map() method.
It removes ~100 lines to subprocess, which is always nice.
msg202234 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-11-05 19:40
Here's an updated patch with a better logic: in the previous version - based on current poll-based implementation, the FD was inferred from the event (i.e. read ready -> stdout/stderr, write ready -> stderr). The new version directly checks the ready file object instead. I also added an extra safety in case an unknown FD is returned (which should never happen).
msg202241 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-05 20:36
subprocess_selectors-3.diff looks good to me (this patch does not
remove any test :-))

I created the issue #19506 for the memoryview optimization.
msg202441 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-11-08 18:55
New changeset 71b618f0c8e9 by Charles-François Natali in branch 'default':
Issue #18923: Update subprocess to use the new selectors module.
http://hg.python.org/cpython/rev/71b618f0c8e9
History
Date User Action Args
2013-11-09 06:48:46neologixsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2013-11-08 18:55:45python-devsetnosy: + python-dev
messages: + msg202441
2013-11-05 20:36:33vstinnersetmessages: + msg202241
2013-11-05 19:40:37neologixsetfiles: + subprocess_selectors-3.diff
nosy: + pitrou
messages: + msg202234

2013-11-01 10:37:03neologixsetfiles: + subprocess_selectors-2.diff
2013-10-30 19:39:47neologixsetfiles: + subprocess_selectors-1.diff

messages: + msg201774
2013-10-05 14:42:42neologixsetfiles: + subprocess_selectors.diff

dependencies: + selectors: add keys() method

keywords: + patch, needs review
nosy: + neologix
messages: + msg198989
stage: patch review
2013-09-05 13:08:38giampaolo.rodolasetmessages: + msg196995
2013-09-05 12:42:56giampaolo.rodolasetnosy: + giampaolo.rodola
2013-09-04 18:44:54vstinnersetmessages: + msg196937
2013-09-04 18:41:35vstinnercreate