Issue12156
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2011-05-23 10:37 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
test_multi_sem.diff | neologix, 2011-11-19 13:24 | review | ||
test_multi_sem-1.diff | neologix, 2011-11-19 17:14 | review |
Messages (10) | |||
---|---|---|---|
msg136606 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-05-23 10:40 | |
http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%203.x/builds/1819/steps/test/logs/stdio [180/355] test_multiprocessing Timeout (1:00:00)! Thread 0x28401d00: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 354 in _send File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 382 in _send_bytes File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 216 in send File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 104 in dispatch File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 746 in _connect File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 758 in _callmethod File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 986 in acquire File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 659 in f File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28401040: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 364 in _recv File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 385 in _recv_bytes File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 260 in recv File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 762 in _callmethod File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/managers.py", line 986 in acquire File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 738 in test_notify_all File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 407 in _executeTestPart File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 462 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 514 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1103 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1191 in _run_suite File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1217 in run_unittest File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 2164 in test_main File "./Lib/test/regrtest.py", line 1048 in runtest_inner File "./Lib/test/regrtest.py", line 842 in runtest File "./Lib/test/regrtest.py", line 666 in main File "./Lib/test/regrtest.py", line 1623 in <module> program finished with exit code 1 elapsedTime=7790.461967 Note: many other tests fail on this buildbot (because of file permissions?) which looks to be ill. |
|||
msg139743 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-07-04 09:19 | |
Another recent hang: http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%207.2%203.x/builds/1951/steps/test/logs/stdio [296/356/3] test_multiprocessing Timeout (1:00:00)! Thread 0x28402be0: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 376 in _handle_results File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28402f10: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 335 in _handle_tasks File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x284039b0: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 326 in _handle_workers File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28402470: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28402cf0: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28401bf0: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28404780: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 102 in worker File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28404560: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 409 in _recv File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 430 in _recv_bytes File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 273 in recv File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 376 in _handle_results File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28401e10: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 237 in wait File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/queue.py", line 185 in get File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 335 in _handle_tasks File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Thread 0x28402360: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/pool.py", line 326 in _handle_workers File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 690 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 737 in _bootstrap_inner File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/threading.py", line 710 in _bootstrap Current thread 0x28401040: File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 409 in _recv File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 434 in _recv_bytes File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/connection.py", line 239 in recv_bytes File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 1512 in test_connection File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 407 in _executeTestPart File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 462 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/case.py", line 514 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 105 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/suite.py", line 67 in __call__ File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/unittest/runner.py", line 168 in run File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1259 in _run_suite File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/support.py", line 1285 in run_unittest File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 2228 in test_main File "./Lib/test/regrtest.py", line 1070 in runtest_inner File "./Lib/test/regrtest.py", line 861 in runtest File "./Lib/test/regrtest.py", line 669 in main File "./Lib/test/regrtest.py", line 1648 in <module> *** Error code 1 |
|||
msg144840 - (view) | Author: Charles-François Natali (neologix) * | Date: 2011-10-03 21:18 | |
test_multiprocessing frequently hangs on FreeBSD < 8 buildbots, and this probably has to do with the limit on the max number of POSIX semaphores: """ ====================================================================== ERROR: test_notify_all (test.test_multiprocessing.WithProcessesTestCondition) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/test/test_multiprocessing.py", line 777, in test_notify_all cond = self.Condition() File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/__init__.py", line 189, in Condition return Condition(lock) File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py", line 198, in __init__ self._lock = lock or RLock() File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py", line 172, in __init__ SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1) File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/multiprocessing/synchronize.py", line 75, in __init__ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 23] Too many open files in system """ There are probably dangling semaphores, since the test doesn't use that much POSIX semaphores. Either way, we can't do much about it... |
|||
msg144861 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-10-04 07:14 | |
> "OSError: [Errno 23] Too many open files in system" Yes, see issue #10348. |
|||
msg147941 - (view) | Author: Charles-François Natali (neologix) * | Date: 2011-11-19 13:24 | |
Alright, I got tired of seeing the FreeBSD buildbots consistently choke on test_multiprocessing, so here's a simple patch that skips the test if the OS doesn't support a reasonable number (30) of semaphores. |
|||
msg147953 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-11-19 16:46 | |
Testing os.sysconf("SC_SEM_NSEMS_MAX") value is maybe better than creating 30 semaphores. See the function added by the changeset 746143ec1f60 (fixing #10798) is maybe faster than creating 30 semaphores. Value on the FreeBSD 7.2 buildbot: >>> import os >>> os.sysconf("SC_SEM_NSEMS_MAX") 30 |
|||
msg147959 - (view) | Author: Charles-François Natali (neologix) * | Date: 2011-11-19 17:14 | |
> Testing os.sysconf("SC_SEM_NSEMS_MAX") value is maybe better than > creating 30 semaphores. Yeah, I thought about that, but the problem is that it doesn't take into account the number of semaphores already allocated: so, for example, if sysconf() returns 50 but you already have 30 allocated semaphores, the test will fail with ENOFILE. But if we consider that most buildbots don't have many semaphores allocated (and in particular no dangling semaphores), and that FreeBSD 8 has a limit substentially higher than 30 (ideally above 256), then I guess we could use this. Patch attached. |
|||
msg148065 - (view) | Author: STINNER Victor (vstinner) * | Date: 2011-11-21 17:20 | |
I tried test_multi_sem-1.diff on "x86 FreeBSD 7.2 3.x" buildbot: the patch works as expected, the test suite is skipped. A minor nit: can you write the number of semaphore in the skip message? Except of this nit, the patch looks good to me (and I prefer test_multi_sem-1.diff over test_multi_sem.diff). |
|||
msg148132 - (view) | Author: Roundup Robot (python-dev) | Date: 2011-11-22 17:56 | |
New changeset ed2d3680f489 by Charles-François Natali in branch '2.7': Issue #12156: Skip test_multiprocessing on systems which don't support enough http://hg.python.org/cpython/rev/ed2d3680f489 New changeset 311bb5e65b02 by Charles-François Natali in branch '3.2': Issue #12156: Skip test_multiprocessing on systems which don't support enough http://hg.python.org/cpython/rev/311bb5e65b02 New changeset 439ac346b20b by Charles-François Natali in branch 'default': Issue #12156: Skip test_multiprocessing on systems which don't support enough http://hg.python.org/cpython/rev/439ac346b20b |
|||
msg148171 - (view) | Author: Charles-François Natali (neologix) * | Date: 2011-11-23 08:09 | |
Alright, the test is now skipped when the system doesn't support enough POSIX semaphores. I'm closing, we can still reopen in case another similar problem pops up (on other OS of course). |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:57:17 | admin | set | github: 56365 |
2011-11-23 08:09:00 | neologix | set | status: open -> closed resolution: fixed messages: + msg148171 stage: resolved |
2011-11-22 17:56:55 | python-dev | set | nosy:
+ python-dev messages: + msg148132 |
2011-11-21 17:20:53 | vstinner | set | messages: + msg148065 |
2011-11-19 17:14:59 | neologix | set | files:
+ test_multi_sem-1.diff messages: + msg147959 |
2011-11-19 16:46:26 | vstinner | set | messages: + msg147953 |
2011-11-19 13:24:51 | neologix | set | files:
+ test_multi_sem.diff keywords: + patch messages: + msg147941 |
2011-10-04 07:14:50 | vstinner | set | messages: + msg144861 |
2011-10-03 21:18:05 | neologix | set | messages: + msg144840 |
2011-07-04 09:19:09 | vstinner | set | messages: + msg139743 |
2011-05-23 10:40:54 | vstinner | set | title: test_multiprocessing -> test_multiprocessing.test_notify_all() timeout (1 hour) on FreeBSD 7.2 nosy: + pitrou, neologix messages: + msg136606 versions: + Python 3.3 components: + Library (Lib) |
2011-05-23 10:37:58 | vstinner | create |