I noticed sometimes regrtest.py fails in test_multiprocessing.py
(test_connection) on win2000.

I could not reproduce error by invoking test_multiprocessing alone, but
finally I could do it by incresing 'really_big_msg' to 32MB or more.

I attached reproducable code. I don't know why this happens yet.

This is traceback when run reproducable.py.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "e:\python-dev\trunk\lib\multiprocessing\forking.py", line 341,
in main
    prepare(preparation_data)
  File "e:\python-dev\trunk\lib\multiprocessing\forking.py", line 456,
in prepar
e
    '__parents_main__', file, path_name, etc
  File "reproducable.py", line 20, in <module>
    conn.send_bytes(really_big_msg)
IOError: [Errno 1450] Insufficient system resources complete the
requested service to exist.

After googling, ERROR_NO_SYSTEM_RESOURCES seems to happen
when one I/O size is too large.

And in Modules/_multiprocessing/pipe_connection.c, conn_send_string is
implemented with one call WriteFile(). Maybe this should be devided into
some reasonable sized chunks for several WriteFile() calls?

I'll try to work on a patch for this, but the reproduce.py script seems
to spawn dozens of sub-interpreters right now when run with trunk
(python 2.7) on win32

John, can you try this on trunk:

from multiprocessing import *

latin = str

SENTINEL = latin('')

def _echo(conn):
    for msg in iter(conn.recv_bytes, SENTINEL):
        conn.send_bytes(msg)
    conn.close()

conn, child_conn = Pipe()

p = Process(target=_echo, args=(child_conn,))
p.daemon = True
p.start()

really_big_msg = latin('X') * (1024 * 1024 * 32)
conn.send_bytes(really_big_msg)
assert conn.recv_bytes() == really_big_msg

conn.send_bytes(SENTINEL)                          # tell child to quit
child_conn.close()

Really? Hmm weird...
I'm using Win2000, maybe are you using newer OS?
Or maybe larger data is needed. This guy says error occurs around 200MB.
(This is async IO though)
>http://www.gamedev.net/community/forums/topic.asp?topic_id=382135

If this happens only on my machine, maybe you can close this entry as
"works for me".

Ah, I forgot this. Process#set_daemon doesn't exist on trunk, I had to
use "p.daemon = True" instead.

John, try this new version

Latest version works -- question is why prior versions spawned many 
subprocesses.  It's really another bug because prior version wasn't 
hitting the write length limit.

The if __name__ clause is actually well documented, see:
http://docs.python.org/library/multiprocessing.html#windows

It turns out that the original reproduce.py deadlocks if the pipe buffer
is smaller than message size -- even with a fix to the bug.  Patch to
fix is coming soon.

Attached is a patch, though I have mixed feelings about it.  The OS
error can still occur even if a smaller amount is written in each
WriteFile call; I think an internal OS buffer fills up and the error is
returned if that buffer is full because the other process hasn't read
yet.  The patch just ignores ERROR_NO_SYSTEM_RESOURCES and writes again.
 I don't know though if ERROR_NO_SYSTEM_RESOURCES can mean something
else is wrong and the write will never succeed.  The message is also
broken up into 32K parts and a recv_bytes on the other end must be
called multiple times to read it all.

The patch is one option.  Another might be to let the application decide
to continue or not and essentially treat the pipes as nonblocking.

I've been thinking about this a bit, and I think raising an exception and 
returning the amount of bytes read makes more sense then just hiding 
it/eating the errors. Explicit > Implicit in this case, at lease doing 
this gives the controller a method of reacting.

Looking into this a bit more and reading the documentation (sorry, I 
picked this up because I know something about win32 and not because I 
know multiprocessing), it looks like a connection is supposed to be 
message oriented and not byte oriented so that a recv() should return 
what is sent in a single send().  This is like how Queue works in the 
threading case.  Note that I think the method signature when using the 
dummy.connection differ when using pipe_connection and that the two 
differ in what happens when several send_bytes's occur before a recv_bytes

I'm currently leaning toward essentially leaving the current behavior 
(and documenting it) though maybe with a better exception and 
documenting that large byte arrays can't be sent through the pipe. 
What's still an issue is if a pickle ends up being too large.

On Wed, Apr 1, 2009 at 2:45 PM, John Ehresman <report@bugs.python.org> wrote:
>
> John Ehresman <jpe@wingware.com> added the comment:
>
> Looking into this a bit more and reading the documentation (sorry, I
> picked this up because I know something about win32 and not because I
> know multiprocessing), it looks like a connection is supposed to be
> message oriented and not byte oriented so that a recv() should return
> what is sent in a single send().  This is like how Queue works in the
> threading case.  Note that I think the method signature when using the
> dummy.connection differ when using pipe_connection and that the two
> differ in what happens when several send_bytes's occur before a recv_bytes
>
> I'm currently leaning toward essentially leaving the current behavior
> (and documenting it) though maybe with a better exception and
> documenting that large byte arrays can't be sent through the pipe.
> What's still an issue is if a pickle ends up being too large.
>

I think I'm fine with this as well, let's documents/add an exception -
as for the pickle being too large, I think for now we're safe
documenting it

New patch which raises ValueError if WriteFile fails with
ERROR_NO_SYSTEM_RESOURCES.  I wasn't able to reliably write a test since
putting the send_bytes in a try block seems to allow the call succeed. 
This is probably OS, swap file size, and timing dependent.

Patch applied in r71036 on python-trunk