New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_threaded_import fails sporadically on Windows and FreeBSD #59804
Comments
[111/369] test_threaded_import
test_circular_imports (test.test_threaded_import.ThreadedImportTests) ... Exception in thread Thread-586:
Traceback (most recent call last):
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/threading.py", line 639, in _bootstrap_inner
self.run()
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/threading.py", line 596, in run
self._target(*self._args, **self._kwargs)
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_threaded_import.py", line 194, in import_ab
import A
ImportError: No module named 'A'
Exception in thread Thread-587:
Traceback (most recent call last):
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/threading.py", line 639, in _bootstrap_inner
self.run()
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/threading.py", line 596, in run
self._target(*self._args, **self._kwargs)
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_threaded_import.py", line 197, in import_ba
import B
ImportError: No module named 'B' FAIL ====================================================================== Traceback (most recent call last):
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_threaded_import.py", line 205, in test_circular_imports
self.assertEqual(set(results), {'a', 'b'})
AssertionError: Items in the second set but not the first:
'a'
'b' |
The buildbot is running inside kvm on a heavily loaded machine. Perhaps |
Could you try to run the tests manually after having upped said timeouts? (look for "sleep" in the test file) |
Antoine Pitrou <report@bugs.python.org> wrote:
No luck there: The tests pass unmodified (100 times with the -F option). |
You could try "-F -j16" or something similar. |
With -F -j16 I get these failures both on FreeBSD and Linux, but not ====================================================================== Traceback (most recent call last):
File "/home/stefan/hg/cpython/Lib/test/test_threaded_import.py", line 131, in test_parallel_meta_path
self.check_parallel_module_init()
File "/home/stefan/hg/cpython/Lib/test/test_threaded_import.py", line 120, in check_parallel_module_init
self.assertFalse(errors)
AssertionError: [AttributeError("'module' object has no attribute 'randrange'",)] is not false |
Reproduced by another FreeBSD bot: |
The FreeBSD machine decided to hang for 1h: If I'm running the tests manually, they take over half an hour and I get: ====================================================================== Traceback (most recent call last):
File "/usr/home/stefan/cpython/Lib/test/test_threaded_import.py", line 133, in test_parallel_meta_path
self.assertEqual(finder.x, finder.numcalls)
AssertionError: 41 != 42 Ran 6 tests in 2090.042s [I hope this is not a threading bug in FreeBSD that appears under |
I see that in the following build test_threading also fails, so I ====================================================================== Traceback (most recent call last):
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/lock_tests.py", line 531, in test_waitfor_timeout
self.assertEqual(len(success), 1)
AssertionError: 0 != 1 |
Did something change recently on that machine? It's supposed to be a |
Nothing changed on the buildbot. -- It appears that FreeBSD is unable to Since Linux has no problems whatsoever, I'm inclined to think that's I've yet to check if the other problem in test_threading is related. |
Maybe. If you believe it's so, I guess the patch can be committed. |
The extreme slowness with a low switch interval is already present in The other FreeBSD 9.0 bots don't seem to have that problem, so I'm not FWIW, http://buildbot.python.org/all/buildslaves/koobs-freebsd runs in |
One of the Windows bots has the same failure in test_parallel_meta_path: http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/5643/steps/test/logs/stdio |
I'm convinced now that some systems just can't handle a low switch http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/5642/steps/test/logs/stdio ====================================================================== Traceback (most recent call last):
File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_concurrent_futures.py", line 71, in tearDown
self.assertLess(dt, 60, "synchronization issue: test lasted too long")
AssertionError: 152.02985382080078 not less than 60 : synchronization issue: test lasted too long That test also uses sys.setswitchinterval(1e-6). |
Ok, let's bump it to 1e-5 then. As for the parallel_meta_path failure, it could be a genuine issue |
The Lion bot also hangs in test_threading: http://buildbot.python.org/all/builders/AMD64%20Lion%203.x/builds/383/steps/test/logs/stdio At least the test_threading failures seem to be recent. The test_threaded_import *slowness* (40 min!) on FreeBSD/KVM with low switch interval goes back a long way. I checked revisions way before |
Stefan, do you plan to commit this patch (or a similar one)? The FreeBSD buildbot is crashing again. |
I was hesitating because the other FreeBSD bots don't have that problem. http://buildbot.python.org/all/buildslaves/koobs-freebsd-clang seems to be very stable. I could take down FreeBSD/kvm altogether. Is Python supposed to work with gil_interval = 1 or is some |
Well, if the test is just running too slow, changing the switch interval
I don't know. It's not supposed to be widely tunable, though. Most |
OK, I'll commit the patch soon. |
New changeset 9b40d018e4cf by Stefan Krah in branch 'default': |
OS X 10.4 (and presumably earlier versions) also does not handle the low switchinterval gracefully. Monitoring system activity and running with -v, test_threaded_import does seem to progress but *very* slowly (I quit after 45 minutes with the test only partially complete) at 100% CPU (mainly sys time). 10.5 and later systems are fine. Here's a patch that extends the FreeBSD solution to OS X 10.4 systems. |
test_threaded_import also takes around 5 min on Windows. I think |
For once a Linux bot shows the failure that is the topic of this ====================================================================== Traceback (most recent call last):
File "/var/lib/buildslave/3.x.murray-gentoo-wide/build/Lib/test/test_threaded_import.py", line 205, in test_circular_imports
self.assertEqual(set(results), {'a', 'b'})
AssertionError: Items in the second set but not the first:
'b'
'a' |
Hmm, so there may be a small race condition after all? The problem is that I can't reproduce, on a non-debug build as on that |
I can't reproduce it either even on the machine that magically caught FWIW, the Gentoo bot also had a completely isolated segfault in Antoine, are you OK with setting the switch interval to 1e-5 for |
Yes, I'm ok with it. |
New changeset 484a4b9349af by Stefan Krah in branch '3.3': |
Is this still an issue? |
Well, there was a test_circular_imports() failure on (I think) the |
Not that it helps much, but here are recent failures on http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/5938/steps/test/logs/stdio http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/5936/steps/test/logs/stdio |
The occasional test_parallel_meta_path failure is really weird. The sys.meta_path module finding in _bootstrap.py is protected by the global import lock, so there shouldn't be any failure here. I'm starting to suspect some issue in the lock code itself. |
New changeset 1ee9cf5a5351 by Antoine Pitrou in branch '3.3': New changeset 7cfca9ede861 by Antoine Pitrou in branch 'default': |
New changeset 67d6bb557d67 by Antoine Pitrou in branch '3.3': New changeset 30c7cc62303a by Antoine Pitrou in branch 'default': |
Another recent instance of the parallel meta path failure: http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/7740/steps/test/logs/stdio |
A recent failure. As usual, the test passed at the second run. ====================================================================== Traceback (most recent call last):
File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/test_threaded_import.py", line 135, in test_parallel_meta_path
self.assertEqual(finder.x, finder.numcalls)
AssertionError: 89 != 90 Re-running test 'test_threaded_import' in verbose mode |
I've been experimenting with setting up a Windows 8.1 buildbot, and found this ticket after finding a problem with test_threaded_import, testing against the 3.4 branch. I seem to be have a low syscheckinterval issue similar to that discussed here on some platforms, though I run into it sooner than 0.00001. If I change the syscheckinterval adjustment to 0.001 the tests run in about 4s. Just slightly below that, 0.0009, it can take well over an hour when run manually, always getting killed due to a timeout when running the buildbot test batch file. Each use of check_module_parallel_init in the test takes 20-30 minutes. During this time the CPU remains pegged at 100%. I don't see any additional slow-down between 0.0009 and the default of 0.00001, so it feels more like crossing a threshold somewhere around 1ms than scaling equally around and through that point. While the machine is not tremendously powerful (it's an Azure VM - single core ~2GHz), everything else is working well and build times and the remainder of the full test suite run in a reasonable time. -- David |
No real activity on this issue. Since the issue is sporadic and I'm not interested to work on it, I just close the issue as out of date. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: