Title: Multiprocessing: Expose underlying pipe in queues
Type: enhancement Stage:
Components: Extension Modules Versions: Python 3.2
Status: open Resolution:
Dependencies: Superseder:
Assigned To: jnoller Nosy List: TarkaSteve, anacrolix, andresfreund, asksol, barcc, forest, jmehnle, jnoller, roudkerk
Priority: low Keywords:

Created on 2008-09-11 01:13 by TarkaSteve, last changed 2014-02-27 22:38 by jmehnle.

Messages (6)
msg73001 - (view) Author: Steve Smith (TarkaSteve) Date: 2008-09-11 01:13
Both Connection and Pipe objects expose their underlying file
descriptors, which is useful for accessing them in an event-driven
manner.  However the higher-level Queue does not make the underlying
pipe available; to get at them you must access private member attributes
which is fragile.  It would be good to have a well-defined API for
accessing either the pipe objects or the underlying FDs.
msg73019 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-11 13:01
Steve, I'm a little nervous about exposing the underlying Queue pipes in 
an official API simply due to the fact that it is an advanced use case, 
and we do want to keep the API simple, and relatively close to the core 
Queue implementation.

Do you have an example use-case/code? This will not be making it into 
python 2.6/3.0 - it will need to wait until 2.7 and 3.1 at very least.
msg73069 - (view) Author: Steve Smith (TarkaSteve) Date: 2008-09-12 00:13
Hi Jesse,

The use-case I had is mind is enabling asyncronous (i.e. select() style)
notification of data being available on the queue, which is more elegant
(and efficient) than polling with get().  Example code showing how this
works with GTK is available here:

I understand wanting to keep the API simple, however the underlying
posix-level file-descriptors for pipes and connections are already
exposed via fileno(); the actual API change would just be a matter of
changing '_reader' and '_writer' to 'reader' and 'writer' (implicitly
guaranteeing their consistency).

Would it help if I attached a patch showing the proposed change to the
module?  I realise the API is frozen now but it would be nice to have
this further down the line.
msg85168 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-04-02 05:36
Can you please provide an example w.r.t to how you would handle the case 
where poll()/recv returns partial bytes of the object instead of a full 
msg107371 - (view) Author: Andres Freund (andresfreund) Date: 2010-06-09 06:34
As soon as some bytes are signalled as being available one can simply do a normal get(). I don't really see the problem here?
Sure, the get() might not be completely non-blocking (especially if the transferred event is more than the size of a pipe-buffer) but I have a hard time seing that as a problem as that should be both rare and only last a short time.

My personal use-case is being able to efficiently wait for evens from different queues - using the standard api one currently can only do that by busy looping...

The biggest thing I see where you have to be careful here is some stomping herd phenomenon you will get into if you have multiple readers doing a poll().
Namely *all* off those processes will awake and run into .get() which isnt exactly nice, but thats hardly solvable on python level.
msg133358 - (view) Author: Matt Joiner (anacrolix) Date: 2011-04-09 00:13
I look forward to this, or something similar. Inspiration can be taken from Golangs's select behaviour on channels.

select {
case i1 = <-c1:
	print("received ", i1, " from c1\n")
case c2 <- i2:
	print("sent ", i2, " to c2\n")

	print("no communication\n")
Date User Action Args
2014-02-27 22:38:01jmehnlesetnosy: + jmehnle
2012-12-26 07:39:34barccsetnosy: + barcc
2011-04-09 00:13:27anacrolixsetmessages: + msg133358
2011-04-09 00:04:15anacrolixsetnosy: + anacrolix
2011-01-21 20:45:30forestsetnosy: + forest
2010-08-31 10:56:54asksolsetnosy: + asksol
2010-08-09 04:41:11terry.reedysetversions: + Python 3.2, - Python 3.1, Python 2.7
2010-06-09 06:34:06andresfreundsetnosy: + andresfreund
messages: + msg107371
2009-04-02 05:36:15jnollersetmessages: + msg85168
2009-01-22 19:20:39jnollersetpriority: low
2008-09-12 00:13:22TarkaStevesetmessages: + msg73069
2008-09-11 13:01:54jnollersetnosy: + roudkerk
messages: + msg73019
versions: + Python 3.1, Python 2.7, - Python 2.6
2008-09-11 02:00:06benjamin.petersonsetassignee: jnoller
nosy: + jnoller
2008-09-11 01:13:58TarkaStevecreate