classification
Title: test_context() of test_multiprocessing_spawn hangs on "x86 Gentoo Non-Debug 3.x" buildbot
Type: Stage:
Components: Tests Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: neologix, sbt, vstinner
Priority: normal Keywords: patch

Created on 2013-11-12 22:06 by vstinner, last changed 2015-03-18 12:46 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
multiprocessing_faulthandler_timeout.patch vstinner, 2013-11-18 09:02 review
Messages (5)
msg202724 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-12 22:06
test_multiprocessing_spawn.test_context() hanged on x86 Gentoo Non-Debug 3.x:

    def test_context(self):
        if self.TYPE == 'processes':
            L = list(range(10))
            expected = [sqr(i) for i in L]
            with multiprocessing.Pool(2) as p:
                r = p.map_async(sqr, L)
                self.assertEqual(r.get(), expected)   <=== HERE === 
            self.assertRaises(ValueError, p.map_async, sqr, L)

http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%203.x/builds/5424/steps/test/logs/stdio

[ 35/385] test_multiprocessing_spawn
Timeout (1:00:00)!
Thread 0xb27fdb40 (most recent call first):
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/connection.py", line 384 in _recv
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/connection.py", line 407 in _recv_bytes
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/connection.py", line 251 in recv
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 415 in _handle_results
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 869 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 921 in _bootstrap_inner
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 889 in _bootstrap

Thread 0xb2ffeb40 (most recent call first):
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 290 in wait
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/queue.py", line 167 in get
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 371 in _handle_tasks
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 869 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 921 in _bootstrap_inner
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 889 in _bootstrap

Thread 0xb37ffb40 (most recent call first):
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 362 in _handle_workers
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 869 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 921 in _bootstrap_inner
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 889 in _bootstrap

Thread 0xb417fb40 (most recent call first):
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/connection.py", line 384 in _recv
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/connection.py", line 407 in _recv_bytes
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/connection.py", line 251 in recv
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 415 in _handle_results
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 869 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 921 in _bootstrap_inner
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 889 in _bootstrap

Thread 0xb520fb40 (most recent call first):
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 290 in wait
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/queue.py", line 167 in get
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 371 in _handle_tasks
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 869 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 921 in _bootstrap_inner
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 889 in _bootstrap

Thread 0xb5a10b40 (most recent call first):
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 362 in _handle_workers
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 869 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 921 in _bootstrap_inner
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 889 in _bootstrap

Thread 0xb76196c0 (most recent call first):
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 290 in wait
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/threading.py", line 553 in wait
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 585 in wait
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/multiprocessing/pool.py", line 588 in get
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/_test_multiprocessing.py", line 1786 in test_context
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/case.py", line 571 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/case.py", line 610 in __call__
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/suite.py", line 117 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/suite.py", line 79 in __call__
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/suite.py", line 117 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/suite.py", line 79 in __call__
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/suite.py", line 117 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/suite.py", line 79 in __call__
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/unittest/runner.py", line 168 in run
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/support/__init__.py", line 1661 in _run_suite
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/support/__init__.py", line 1695 in run_unittest
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/regrtest.py", line 1275 in <lambda>
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/regrtest.py", line 1276 in runtest_inner
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/regrtest.py", line 965 in runtest
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/regrtest.py", line 761 in main
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/regrtest.py", line 1560 in main_in_temp_cwd
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/__main__.py", line 3 in <module>
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/runpy.py", line 73 in _run_code
  File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/runpy.py", line 160 in _run_module_as_main
make: *** [buildbottest] Error 1
msg203260 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-18 08:47
See issue #12413 which may help to analyze this hang.
msg203262 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-18 09:02
multiprocessing_faulthandler_timeout.patch: Dump the traceback of all Python threads in a child process if it takes more than 10 minutes to succeed. On the buildbot, the process hangs more than 1 hour. I guess that 10 minutes in enough for a single unit test.

The patch adds a private attribute "_faulthandler_timeout" to Popen classes, but also a "_faulthandler_timeout" parameter to spawn_main(). If you think that it is really too ugly, the patch may be reverted after the bug is analyzed. Or would it be possible to pass a command to "prepare" a child process, like preexec_fn of the subprocess module?
msg203275 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-11-18 10:36
I don't think the patch to the _test_multiprocessing will work.  It defines cls._Popen but I don't see how that would be used by cls.Pool to start the processes.

I will have a think about a fix.
msg238423 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 12:46
I didn't see this sporadic failure recently. The buildbot is now offline. I close the issue.
History
Date User Action Args
2015-03-18 12:46:37vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg238423
2013-11-18 10:36:42sbtsetmessages: + msg203275
2013-11-18 09:02:43vstinnersetfiles: + multiprocessing_faulthandler_timeout.patch

nosy: + neologix
messages: + msg203262

keywords: + patch
2013-11-18 08:47:14vstinnersetmessages: + msg203260
2013-11-18 08:46:23vstinnersettitle: test_multiprocessing_spawn hangs -> test_context() of test_multiprocessing_spawn hangs on "x86 Gentoo Non-Debug 3.x" buildbot
2013-11-12 22:08:05vstinnersettitle: test_multiprocessing_spawn -> test_multiprocessing_spawn hangs
2013-11-12 22:06:31vstinnercreate