It seems the old pyprocessing (http://pyprocessing.berlios.de/) can do 
some things that the new multiprocessing package can not; 
sending/receiving connection objects for one.

This is a quite handy functionality, so it would be nice if it were 
reintroduced.

Also, the error message isn't very helpful.

Failing test below.


$ python2.6 pipetest2.py 
asdf
Traceback (most recent call last):
  File "a.py", line 10, in <module>
    print c1.recv()
TypeError: Required argument 'handle' (pos 1) not found

$ PYTHONPATH=processing-0.52-py2.5-macosx-10.5-i386.egg python2.5 
pipetest2.py 
asdf
Connection(handle=5)

$ PYTHONPATH=multiprocessing-2.6.0.2-py2.5-macosx-10.5-i386.egg 
python2.5 pipetest2.py 
asdf
Traceback (most recent call last):
  File "pipetest2.py", line 10, in <module>
    print c1.recv()
TypeError: function takes at least 1 argument (0 given)

$ uname -a
Darwin midori.local 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 
17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386

$ cat pipetest2.py 
try:
	from multiprocessing import Pipe
except ImportError:
	from processing import Pipe

c1, c2 = Pipe(duplex=False)
c2.send('asdf')
print c1.recv()
c2.send(c1)
print c1.recv()

thanks for filing this. I'll need to compare the two code bases and figure 
out why it's either regressed, or Richard removed it prior to the 
integration.

Before I can logically support this, I need a clear use case that supports 
the idea that this should be supported in the current version of 
multiprocessing.

A typical use case would be for a server to receive a connection, and
then send that connection over to another process that does the actual
work. This used to work with pyprocessing, and the support seems to be
available in multiprocessing.c -> multiprocessing_sendfd, but not used.

And to be clear, I have enabled connection pickling by issuing:

multiprocessing.allow_connection_pickling()

When reverting this commit stuff works:

http://svn.python.org/view/python/trunk/Lib/multiprocessing/reduction.py?r1=64257&r2=65016

Ehm.. completly broken url in prev message.. Revision 65016, "Apply
Amaury's patch to multiprocessing for issue 3125, removes the copy_reg
and replaces it with ForkingPickler.register(), which should resolve the
conflict with the global registry/ctypes" is what I'm refering to.
Without this patch, the reducers/rebuilders are properly registered in
the pickler that connection.h (srsly, code in header files?) later uses.

Wanted to quickly comment here, as I'm dealing with this issue as well, that I did find a workaround for avoiding it as far back as 2.6 (and it's not "don't pass a Pipe through a Pipe")

multiprocessing.reduction can already do this, though I don't entirely know why this isn't automatically done if it's a connection object.

>>> from multiprocessing import Pipe, reduction
>>> i, o = Pipe()
>>> reduced = reduction.reduce_connection(i)
>>> newi = reduced[0](*reduced[1])
>>> newi.send("hi")
>>> o.recv()
'hi'
>>> 

The reduced[0](*reduced[1]) line is actually calling reduction.rebuild_connection, as that function is the first element in the tuple, and the second element is the arguments to be passed to it. I can't seem to find any info on reduction.reduce_connection, so I don't know if this is how this was intended to be handled or not.

P.S. Tested on Win (XP) and Linux (Ubuntu 10.10), so there's no weird windows socket stuff that should go wrong with this.