classification
Title: test_multiprocessing.test_notify_all() timeout (1 hour) on FreeBSD 7.2
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: neologix, pitrou, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2011-05-23 10:37 by vstinner, last changed 2011-11-23 08:09 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
test_multi_sem.diff neologix, 2011-11-19 13:24 review
test_multi_sem-1.diff neologix, 2011-11-19 17:14 review
Messages (10)
msg136606 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-23 10:40
http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%203.x/builds/1819/steps/test/logs/stdio

[180/355] test_multiprocessing
Timeout (1:00:00)!
Thread 0x28401d00:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 354 in _send
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 382 in _send_bytes
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 216 in send
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 104 in dispatch
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 746 in _connect
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 758 in _callmethod
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 986 in acquire
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 659 in f
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28401040:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 364 in _recv
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 385 in _recv_bytes
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 260 in recv
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 762 in _callmethod
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 986 in acquire
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 738 in test_notify_all
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 407 in _executeTestPart
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 462 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 514 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1103 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1191 in _run_suite
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1217 in run_unittest
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 2164 in test_main
  File "./Lib/test/regrtest.py", line 1048 in runtest_inner
  File "./Lib/test/regrtest.py", line 842 in runtest
  File "./Lib/test/regrtest.py", line 666 in main
  File "./Lib/test/regrtest.py", line 1623 in <module>
program finished with exit code 1
elapsedTime=7790.461967

Note: many other tests fail on this buildbot (because of file permissions?) which looks to be ill.
msg139743 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-07-04 09:19
Another recent hang:
http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%203.x/builds/1951/steps/test/logs/stdio

[296/356/3] test_multiprocessing
Timeout (1:00:00)!
Thread 0x28402be0:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 376 in _handle_results
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28402f10:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 335 in _handle_tasks
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x284039b0:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 326 in _handle_workers
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28402470:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28402cf0:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28401bf0:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28404780:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28404560:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 409 in _recv
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 430 in _recv_bytes
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 273 in recv
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 376 in _handle_results
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28401e10:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 335 in _handle_tasks
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Thread 0x28402360:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 326 in _handle_workers
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap

Current thread 0x28401040:
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 409 in _recv
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 434 in _recv_bytes
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 239 in recv_bytes
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 1512 in test_connection
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 407 in _executeTestPart
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 462 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 514 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/runner.py", line 168 in run
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1259 in _run_suite
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1285 in run_unittest
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 2228 in test_main
  File "./Lib/test/regrtest.py", line 1070 in runtest_inner
  File "./Lib/test/regrtest.py", line 861 in runtest
  File "./Lib/test/regrtest.py", line 669 in main
  File "./Lib/test/regrtest.py", line 1648 in <module>
*** Error code 1
msg144840 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-10-03 21:18
test_multiprocessing frequently hangs on FreeBSD < 8 buildbots, and this probably has to do with the limit on the max number of POSIX semaphores:
"""
======================================================================
ERROR: test_notify_all (test.test_multiprocessing.WithProcessesTestCondition)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 777, in test_notify_all
    cond = self.Condition()
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/__init__.py", line 189, in Condition
    return Condition(lock)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py", line 198, in __init__
    self._lock = lock or RLock()
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py", line 172, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 23] Too many open files in system
"""

There are probably dangling semaphores, since the test doesn't use that much POSIX semaphores. Either way, we can't do much about it...
msg144861 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-10-04 07:14
> "OSError: [Errno 23] Too many open files in system"

Yes, see issue #10348.
msg147941 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-11-19 13:24
Alright, I got tired of seeing the FreeBSD buildbots consistently choke on test_multiprocessing, so here's a simple patch that skips the test if the OS doesn't support a reasonable number (30) of semaphores.
msg147953 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-11-19 16:46
Testing os.sysconf("SC_SEM_NSEMS_MAX") value is maybe better than creating 30 semaphores. See the function added by the changeset 746143ec1f60 (fixing #10798) is maybe faster than creating 30 semaphores.

Value on the FreeBSD 7.2 buildbot:

>>> import os
>>> os.sysconf("SC_SEM_NSEMS_MAX")
30
msg147959 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-11-19 17:14
> Testing os.sysconf("SC_SEM_NSEMS_MAX") value is maybe better than
> creating 30 semaphores.

Yeah, I thought about that, but the problem is that it doesn't take into account the number of semaphores already allocated: so, for example, if sysconf() returns 50 but you already have 30 allocated semaphores, the test will fail with ENOFILE.
But if we consider that most buildbots don't have many semaphores allocated (and in particular no dangling semaphores), and that FreeBSD 8 has a limit substentially higher than 30 (ideally above 256), then I guess we could use this.
Patch attached.
msg148065 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-11-21 17:20
I tried test_multi_sem-1.diff on "x86 FreeBSD 7.2 3.x" buildbot: the patch works as expected, the test suite is skipped.

A minor nit: can you write the number of semaphore in the skip message? Except of this nit, the patch looks good to me (and I prefer test_multi_sem-1.diff over test_multi_sem.diff).
msg148132 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-11-22 17:56
New changeset ed2d3680f489 by Charles-François Natali in branch '2.7':
Issue #12156: Skip test_multiprocessing on systems which don't support enough
http://hg.python.org/cpython/rev/ed2d3680f489

New changeset 311bb5e65b02 by Charles-François Natali in branch '3.2':
Issue #12156: Skip test_multiprocessing on systems which don't support enough
http://hg.python.org/cpython/rev/311bb5e65b02

New changeset 439ac346b20b by Charles-François Natali in branch 'default':
Issue #12156: Skip test_multiprocessing on systems which don't support enough
http://hg.python.org/cpython/rev/439ac346b20b
msg148171 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-11-23 08:09
Alright, the test is now skipped when the system doesn't support enough POSIX semaphores.
I'm closing, we can still reopen in case another similar problem pops up (on other OS of course).
History
Date User Action Args
2011-11-23 08:09:00neologixsetstatus: open -> closed
resolution: fixed
messages: + msg148171

stage: resolved
2011-11-22 17:56:55python-devsetnosy: + python-dev
messages: + msg148132
2011-11-21 17:20:53vstinnersetmessages: + msg148065
2011-11-19 17:14:59neologixsetfiles: + test_multi_sem-1.diff

messages: + msg147959
2011-11-19 16:46:26vstinnersetmessages: + msg147953
2011-11-19 13:24:51neologixsetfiles: + test_multi_sem.diff
keywords: + patch
messages: + msg147941
2011-10-04 07:14:50vstinnersetmessages: + msg144861
2011-10-03 21:18:05neologixsetmessages: + msg144840
2011-07-04 09:19:09vstinnersetmessages: + msg139743
2011-05-23 10:40:54vstinnersettitle: test_multiprocessing -> test_multiprocessing.test_notify_all() timeout (1 hour) on FreeBSD 7.2
nosy: + pitrou, neologix

messages: + msg136606

versions: + Python 3.3
components: + Library (Lib)
2011-05-23 10:37:58vstinnercreate