classification
Title: multiprocessing.queues.SimpleQueue is undocumented
Type: Stage: resolved
Components: Documentation, Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eli.bendersky, ezio.melotti, pitrou, python-dev, sandro.tosi, sbt
Priority: normal Keywords: easy, patch

Created on 2011-04-12 15:56 by pitrou, last changed 2012-02-15 22:33 by sandro.tosi. This issue is now closed.

Files
File name Uploaded Description Edit
issue11836-default.patch sandro.tosi, 2011-08-08 13:34 review
Messages (18)
msg133586 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-12 15:56
multiprocessing.queues.SimpleQueue is undocumented and doesn't appear in multiprocessing.__all__.
msg141777 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2011-08-08 13:33
Here's a patch to add SimpleQueue to doc and __all__ .

I didn't document the 'sentinels' argument of SimpleQueue.get() because I got lost into Pipe _pool and can't understand how those sentinels are actually used/useful. Any hint is appreciated :)
msg147579 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-11-13 23:24
Sandro - yep, the sentinels arg is also undocumented in multiprocessing.PipeConnection.recv() and further down the road...
msg147581 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-11-13 23:28
However, sentinels *are* mentioned in the multiprocessing doc, below multiprocessing.Process:

"
sentinel

    A numeric handle of a system object which will become “ready” when the process ends.

    On Windows, this is an OS handle usable with the WaitForSingleObject and WaitForMultipleObjects family of API calls. On Unix, this is a file descriptor usable with primitives from the select module.

    You can use this value if you want to wait on several events at once. Otherwise calling join() is simpler.

    New in version 3.3.
"

From a cursory glance on the implementation in Lib/multiprocessing/connection.py, this is the same sentinel.
msg147617 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-14 16:59
Well, the sentinels argument, right now, is meant to be used internally. I don't think it's a good thing to document it, since I don't think it's a very clean API (I know, I introduced it :-)) - it's just so that concurrent.futures can detect dead processes.
msg147629 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2011-11-14 19:50
> Well, the sentinels argument, right now, is meant to be used 
> internally. I don't think it's a good thing to document it, 
> since I don't think it's a very clean API (I know, I introduced 
> it :-))

Wouldn't a better alternative be to have a wait function which can deal with readable pipe connections and integer handles?

On Unix this would just delegate to select().

On Windows it could work as follows:
* initiate an overlapped read on each connection
* call WaitForMultipleObjects()
* cancel each overlapped read
* continue any read which succeeded but only gave a partial message
* store read messages on associated connection objects

I did start on such a patch.  It worked, but I did not get round to writing tests for it...
msg147633 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-14 20:53
> Wouldn't a better alternative be to have a wait function which can
> deal with readable pipe connections and integer handles?
> 
> On Unix this would just delegate to select().
> 
> On Windows it could work as follows:
> * initiate an overlapped read on each connection
> * call WaitForMultipleObjects()
> * cancel each overlapped read
> * continue any read which succeeded but only gave a partial message
> * store read messages on associated connection objects

This sounds like a good direction to me.
(of course it must also preserve existing functionality;
test_concurrent_futures will check for that)
msg147717 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-11-15 22:42
Then it appears to me that Sandro's patch is good and can be committed.
msg147720 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-15 22:52
> Then it appears to me that Sandro's patch is good and can be committed.

The doc patch is good. However, if you start exposing SimpleQueue at the
top package level, you have to do it in 3.3 only (since that's a new
API), and also mention it somehow in the doc.
msg152929 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-02-09 04:35
Sandro, can you commit, taking Antoine's note into account?
msg153215 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2012-02-12 18:32
Thanks Eli for the heads-up, I had missed Antoine's comment!

Antoine, I'm probably missing something, but SimpleQueue is present in 2.7 and 3.2 too, so why not mention it in the doc for those versions too?
msg153216 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-12 18:34
> Antoine, I'm probably missing something, but SimpleQueue is present in 
> 2.7 and 3.2 too, so why not mention it in the doc for those versions too?

What I mean is that "multiprocessing.SimpleQueue" is a new API, so should be 3.3-only (while "multiprocessing.queues.SimpleQueue" already exists).
msg153219 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2012-02-12 18:50
It's the way all the subclasses are imported into the main module that got me in fault, I think. OK, so if I got it correctly, I should document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 and multiprocessing.SimpleQueue in 3.3 also adding the hunk to __init__ (thus changing the API).

I'd say to document int still near to Queue() and JoinableQueue() - do you agree?
msg153220 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-12 18:59
> It's the way all the subclasses are imported into the main module that
> got me in fault, I think. OK, so if I got it correctly, I should
> document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 and
> multiprocessing.SimpleQueue in 3.3 also adding the hunk to __init__
> (thus changing the API).

Yes.

> I'd say to document int still near to Queue() and JoinableQueue() - do
> you agree?

Yes, too.
msg153225 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2012-02-12 19:14
> OK, so if I got it correctly, I should document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 [...]


s/3.1/3.2/
msg153227 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2012-02-12 19:53
>> OK, so if I got it correctly, I should document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 [...]
>
> s/3.1/3.2/

yeah, just a typo :)
msg153444 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-02-15 22:32
New changeset 3b127a415643 by Sandro Tosi in branch '2.7':
Issue #11836: document multiprocessing.queues.SimpleQueue
http://hg.python.org/cpython/rev/3b127a415643

New changeset fe5eb6d35025 by Sandro Tosi in branch '3.2':
Issue #11836: document multiprocessing.queues.SimpleQueue
http://hg.python.org/cpython/rev/fe5eb6d35025

New changeset bf536b46d7f2 by Sandro Tosi in branch 'default':
Issue #11836: document and expose multiprocessing.SimpleQueue
http://hg.python.org/cpython/rev/bf536b46d7f2
msg153445 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2012-02-15 22:33
Thanks for all you inputs!
History
Date User Action Args
2012-02-15 22:33:46sandro.tosisetstatus: open -> closed
resolution: fixed
messages: + msg153445

stage: patch review -> resolved
2012-02-15 22:32:02python-devsetnosy: + python-dev
messages: + msg153444
2012-02-12 19:53:44sandro.tosisetmessages: + msg153227
2012-02-12 19:14:24eli.benderskysetmessages: + msg153225
2012-02-12 19:13:55eli.benderskysetmessages: - msg153224
2012-02-12 19:13:13eli.benderskysetmessages: + msg153224
2012-02-12 18:59:41pitrousetmessages: + msg153220
2012-02-12 18:50:14sandro.tosisetmessages: + msg153219
2012-02-12 18:34:34pitrousetmessages: + msg153216
2012-02-12 18:32:13sandro.tosisetmessages: + msg153215
2012-02-09 04:35:48eli.benderskysetmessages: + msg152929
2011-11-15 22:52:53pitrousetmessages: + msg147720
2011-11-15 22:42:25eli.benderskysetmessages: + msg147717
2011-11-14 20:53:37pitrousetmessages: + msg147633
2011-11-14 19:50:19sbtsetnosy: + sbt
messages: + msg147629
2011-11-14 16:59:42pitrousetmessages: + msg147617
2011-11-13 23:28:19eli.benderskysetmessages: + msg147581
2011-11-13 23:24:28eli.benderskysetmessages: + msg147579
2011-08-08 13:34:02sandro.tosisetfiles: + issue11836-default.patch
keywords: + patch
2011-08-08 13:33:42sandro.tosisetversions: + Python 2.7, - Python 3.1
nosy: + ezio.melotti, sandro.tosi

messages: + msg141777

stage: needs patch -> patch review
2011-04-12 17:56:44eli.benderskysetnosy: + eli.bendersky
2011-04-12 15:56:24pitroucreate