https://travis-ci.org/python/cpython/jobs/379560387

======================================================================
ERROR: test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/_test_multiprocessing.py", line 4359, in test_ignore
    os.kill(p.pid, signal.SIGUSR1)
ProcessLookupError: [Errno 3] No such process

FWIW, Issue29589 seems to be an older example of this failure.

> FWIW, Issue29589 seems to be an older example of this failure.

It was a failure on Mageia 5 (Linux) at 2017-02-17:

== CPython 3.6.0 (default, Feb 17 2017, 15:26:31) [GCC 4.9.2]
==   Linux-4.4.13-desktop-1.mga5-x86_64-with-mageia-5-Official little-endian
==   hash algorithm: siphash24 64bit
==  cwd: /home/dima/bin/Python-3.6.0/build/test_python_8244
==  encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 [1/1] test_multiprocessing_forkserver
...
======================================================================
ERROR: test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dima/bin/Python-3.6.0/Lib/test/_test_multiprocessing.py", line 3728, in test_ignore
    os.kill(p.pid, signal.SIGUSR1)
ProcessLookupError: [Errno 3] No such process

I am unable to reproduce the issue on Fedora 28 (Linux kernel 4.16.11, glibc 2.27):

I ran 6 jobs in parallel during 10 minutes:

* 2: ./python -m test test_multiprocessing_forkserver -m test_ignore -F
* 2: ./python -m test test_multiprocessing_forkserver -F
* 2: ./python -m test -j16 -r -F

The system load was around 30 which is very high (my CPU has 8 logicial threads, 4 physical cores).

> https://travis-ci.org/python/cpython/jobs/379560387

More info about this failure:

== Linux-4.4.0-112-generic-x86_64-with-debian-jessie-sid little-endian
== CPU count: 48
Run tests in parallel using 4 child processes
(...)
0:07:06 load avg: 24.76 [356/416/3] test_multiprocessing_forkserver failed -- running: test_concurrent_futures (61 sec)

Davin, Antoine: any idea on this bug?

I ran the full test suite and test_multiprocessing_forkserver in parallel in a Trust VM, but I failed to reproduce the bug.

I did a similar test in a Ubuntu Trusty docker container: again, I'm unable to reproduce the bug.

An Ubuntu Trusty docker container is supposed to be as close as possible to Travis CI, except that my laptop doesn't have 48 CPUs.

test_ignore() starts to fail more and more often on Travis CI for an unknown reason.

https://travis-ci.org/python/cpython/jobs/385562187

== CPU count: 48
(...)
Run tests in parallel using 4 child processes
(...)
0:07:38 load avg: 128.04 [415/416/1] test_multiprocessing_forkserver failed -- running: test_concurrent_futures (378 sec)
(...)
======================================================================
ERROR: test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/_test_multiprocessing.py", line 4359, in test_ignore
    os.kill(p.pid, signal.SIGUSR1)
ProcessLookupError: [Errno 3] No such process

https://travis-ci.org/python/cpython/jobs/379560387

Re-running test 'test_multiprocessing_forkserver' in verbose mode
(...)
======================================================================
ERROR: test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/_test_multiprocessing.py", line 4359, in test_ignore
    os.kill(p.pid, signal.SIGUSR1)
ProcessLookupError: [Errno 3] No such process

Python 3.7:

https://travis-ci.org/python/cpython/jobs/385474104

Re-running test 'test_multiprocessing_forkserver' in verbose mode

======================================================================
ERROR: test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/_test_multiprocessing.py", line 4324, in test_ignore
    os.kill(p.pid, signal.SIGUSR1)
ProcessLookupError: [Errno 3] No such process

Python 3.7:

https://travis-ci.org/python/cpython/jobs/385458840

Re-running test 'test_multiprocessing_forkserver' in verbose mode

======================================================================
ERROR: test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/_test_multiprocessing.py", line 4324, in test_ignore
    os.kill(p.pid, signal.SIGUSR1)
ProcessLookupError: [Errno 3] No such process

I added debug traces in PR 7260. test_ignore() failed, but it may be related to my debug traces, since the failure is different.


test_multiprocessing_fork.test_ignore() failure:

test_ignore (test.test_multiprocessing_fork.TestIgnoreEINTR) ...
parent: pid=20159
parent: start child
parent: p=<Process(Process-3, started daemon)>
parent: p.ident=20176
ERROR

======================================================================
ERROR: test_ignore (test.test_multiprocessing_fork.TestIgnoreEINTR)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/_test_multiprocessing.py", line 4389, in test_ignore
    self.assertEqual(conn.recv(), 'ready')
  File "/home/travis/build/python/cpython/Lib/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/travis/build/python/cpython/Lib/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/travis/build/python/cpython/Lib/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError


test_multiprocessing_fork.test_ignore() success:

test_ignore (test.test_multiprocessing_fork.TestIgnoreEINTR) ...
parent: pid=11713
parent: start child
parent: p=<Process(Process-3, started daemon)>
parent: p.ident=28998
[INFO/Process-3] child process calling self.run()
child: pid=28998
child: register SIGUSR1 handler, send ready
child: wait 1234
parent: ready received
parent: first SIGUSR1
child: got SIGUSR1
parent: send 1234
parent: wait 1234
child: send back 1234
child: big blocking send
parent: second SIGUSR1
parent: wait bytes
child: got SIGUSR1
child: done
[INFO/Process-3] process shutting down
[INFO/Process-3] process shutting down
[INFO/Process-3] process exiting with exitcode 0
[INFO/Process-3] process exiting with exitcode 0
parent: join child
parent: done
ok

> I added debug traces in PR 7260. test_ignore() failed, but it may be related to my debug traces, since the failure is different.

https://travis-ci.org/python/cpython/jobs/385958924

Notes:

* test_ignore() creates a daemonic process (it cannot spawn new processes)
* the fork server process is shared between test methods of test_forkserver
* multiprocessing.util._exit_function() terminates all daemon processes

test_multiprocessing_forkserver.test_ignore() failed on Travis CI using PR 7261: --randseed=7474929. The method failed once when run in parallel, and then failed again when re-reun in verbose mode.

> test_multiprocessing_forkserver.test_ignore() failed on Travis CI using PR 7261: --randseed=7474929. The method failed once when run in parallel, and then failed again when re-reun in verbose mode.

https://travis-ci.org/python/cpython/jobs/385986803

0:04:57 load avg: 25.81 [342/415/1] test_multiprocessing_forkserver failed -- running: test_concurrent_futures (72 sec)

I added more traces to PR 7261 and the bug still occurred on Travis CI.

https://travis-ci.org/python/cpython/jobs/385990848

0:05:04 load avg: 42.62 [342/415/1] test_multiprocessing_forkserver failed -- running: test_concurrent_futures (69 sec)
child: pid=19401 ppid=19348
child: send BIG
child: done
test_answer_challenge_auth_failure (test.test_multiprocessing_forkserver.OtherTest) ... ok
test_deliver_challenge_auth_failure (test.test_multiprocessing_forkserver.OtherTest) ... ok
test_closefd (test.test_multiprocessing_forkserver.TestCloseFds) ... ok
test_flags (test.test_multiprocessing_forkserver.TestFlags) ... ok
test_lock (test.test_multiprocessing_forkserver.TestForkAwareThreadLock) ... ok
test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR) ... parent: pid=19342
parent: second SIGUSR1
ERROR

Hum... maybe the child exited, before the parent sent SIGUSR1: the child didn't block on sending 1 MB?

New changeset 5d6c7ed5e340b2311a15f34e968d4bef09c71922 by Victor Stinner in branch 'master':
bpo-33532: Fix multiprocessing test_ignore() (#7262)
https://github.com/python/cpython/commit/5d6c7ed5e340b2311a15f34e968d4bef09c71922

New changeset 9d273344238330d634acb4a7d427a8fb732ae62e by Miss Islington (bot) in branch '3.7':
bpo-33532: Fix multiprocessing test_ignore() (GH-7262)
https://github.com/python/cpython/commit/9d273344238330d634acb4a7d427a8fb732ae62e

New changeset c2870b699eb899f358b843c2d706ce8ca89b2bf4 by Victor Stinner in branch '3.6':
bpo-33532: Fix multiprocessing test_ignore() (GH-7262) (#7266)
https://github.com/python/cpython/commit/c2870b699eb899f358b843c2d706ce8ca89b2bf4

New changeset 82c456fa37d4b87da0d64e574bd8adc665abc4c8 by Victor Stinner in branch '2.7':
bpo-33532: Fix multiprocessing test_ignore() (GH-7265)
https://github.com/python/cpython/commit/82c456fa37d4b87da0d64e574bd8adc665abc4c8

Did your PR fix the issue?

> Did your PR fix the issue?

The bug was that *sometimes* on Travis CI, and only on Travis CI (!?), writing 1 MiB into the multiprocessing pipe didn't block. The bug is really strange because it is only reproduced on the clang Linux job of Travis CI which runs tests in parallel. Not on the Linux gcc which runs tests sequentially in coverage. Moreover, the failure only occurs for a specific order of tests.

You can easily reproduce the issue if you reduce the size of the data written into the pipe at the end of _test_ignore(). If the write (send_bytes) doesn't block, you get the same error.

I'm confident that writing 4 MiB instead of 1 MiB will fix the issue. I saw the test passing with 4 MiB whereas it failed with 1 MiB, when I fixed the test order.

I saw one new test_ignore() failure on Travis CI in my 3.6 PR, whereas 3.6 already uses PIPE_MAX_SIZE for test_ignore().

https://github.com/python/cpython/pull/7315/
https://travis-ci.org/python/cpython/jobs/386637729

Hum, test_ignore() uses PIPE_MAX_SIZE, whereas the test fails on Linux where we use a pair of sockets, no pipes. Maybe we should use SOCK_MAX_SIZE?

Notes on pipe size:

* /proc/sys/fs/pipe-max-size
* bpo-17835: add support.PIPE_MAX_SIZE (4 MiB +1 B)
* bpo-18643: add support.SOCK_MAX_SIZE (16 MiB)

> https://travis-ci.org/python/cpython/jobs/386637729

I added some debug traces:

test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR) ...
socket[fd=4].SO_RCVBUF 212992
socket[fd=4].SO_SNDBUF 212992
socket[fd=7].SO_RCVBUF 212992
socket[fd=7].SO_SNDBUF 212992
test_ignore: PIPE_MAX_SIZE = 4194305

The socket pair uses a buffer of 208 KiB (212,992 B) in both directions and the test sends 4 MiB (4,194,305 B) for a blocking call... and sometimes the send doesn't block.

New changeset 252f6abe0a9430f4ae7588c0cb50a6ff141bebe3 by Victor Stinner in branch 'master':
bpo-33532: Fix test_multiprocessing_forkserver.test_ignore() (GH-7319)
https://github.com/python/cpython/commit/252f6abe0a9430f4ae7588c0cb50a6ff141bebe3

I modified test_ignore() to use support.SOCK_MAX_SIZE, but honestly, I'm not convinced that it will be the issue. I applied my change anyway, just to check if the issue comes from the size, or if it's something else.

New changeset 63fa8db58c349f985d75139e6545cfddf067c07b by Miss Islington (bot) in branch '3.7':
bpo-33532: Fix test_multiprocessing_forkserver.test_ignore() (GH-7319)
https://github.com/python/cpython/commit/63fa8db58c349f985d75139e6545cfddf067c07b

New changeset 1d4be0a65f66c2eecefb8b8bd4d115a82fa61067 by Victor Stinner in branch '2.7':
bpo-33532: Fix test_multiprocessing_forkserver.test_ignore() (GH-7323)
https://github.com/python/cpython/commit/1d4be0a65f66c2eecefb8b8bd4d115a82fa61067

New changeset 64e538bc703e423a04ab435c4eab6b950b8aef7e by Victor Stinner in branch '3.6':
bpo-33532: Fix test_multiprocessing_forkserver.test_ignore() (GH-7322)
https://github.com/python/cpython/commit/64e538bc703e423a04ab435c4eab6b950b8aef7e

It didn't see this super annoying failure recently, so it seems like it has been fixed for real. Great!