New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_multiprocessing_spawn leaked [1, 2, 1] memory blocks on AMD64 Windows8.1 Refleaks 3.7 #77916
Comments
http://buildbot.python.org/all/#/builders/132/builds/154 test_multiprocessing_spawn leaked [1, 2, 1] memory blocks, sum=4 |
When running "python -m test -R 2:3 test_multiprocessing_forkserver" on Windows, I saw some warnings about dangling threads. It may explain this issue. |
bpo-33853 has been marked as a duplicate this bug. |
That's the AMD64 Windows8.1 Refleaks 3.7 buildbot.
Copy of Pablo's message: The test x86 Gentoo Refleaks 3.x test_multiprocessing_spawn leaked [1, 2, 1] memory blocks, sum=4 x86 Gentoo Refleaks 3.7 |
I also created bpo-33966: "test_multiprocessing_spawn.WithProcessesTestPool.test_traceback() leaks 4 handles on Windows". |
I reproduced the issue on the Gentoo Refleak buildbot and I succeeded to bisect up to a single test: test_imap_unordered(). pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3 == Tests result: FAILURE == 1 test failed: Total duration: 6 sec 548 ms |
It doesn't look like a real leak, but more a cache which takes multiple iterations to be fully filled. pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 1:30 == Tests result: FAILURE == 1 test failed: Total duration: 42 sec 490 ms |
Sorry, I forgot to mention that I modified libregrtest to get this output: diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py
index 6724488fcf..a3c50e21e0 100644
--- a/Lib/test/libregrtest/refleak.py
+++ b/Lib/test/libregrtest/refleak.py
@@ -101,7 +101,7 @@ def dash_R(the_module, test, indirect_test, huntrleaks):
failed = False
for deltas, item_name, checker in [
(rc_deltas, 'references', check_rc_deltas),
- (alloc_deltas, 'memory blocks', check_rc_deltas),
+ (alloc_deltas, 'memory blocks', check_fd_deltas),
(fd_deltas, 'file descriptors', check_fd_deltas)
]:
# ignore warmup runs |
It failed for the first time on Ubuntu and then was successful for all the rest of 5-6 runs. I don't know why for the failure run it has load avg as 0.00 and how to get to this stage. # Shell session ➜ cpython git:(master) uname -a
Linux ubuntu-s-1vcpu-1gb-blr1-01 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
➜ cpython git:(master) ./python
Python 3.8.0a0 (heads/master:d824ca7, Jul 3 2018, 06:50:05)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
➜ cpython git:(master) ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3
Run tests sequentially
0:00:00 load avg: 0.00 [1/1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
test_multiprocessing_spawn leaked [2, 2, 1] memory blocks, sum=5
test_multiprocessing_spawn failed == Tests result: FAILURE == 1 test failed: Total duration: 9 sec 221 ms == Tests result: SUCCESS == 1 test OK. Total duration: 8 sec 822 ms |
The bug is random. But the problem is that sometimes, it fails. It must never fail, otherwise the buildbot fails randomly. The Gentoo Refleak buildbot runs multiple tests in parallel and so its system load is high, tests are run slower, making the failure more likely. Anyway, I have a fix! PR 8059. |
bpo-33984 has been marked as a duplicate of this issue: "test_multiprocessing_forkserver leaked [1, 2, 1] memory blocks on x86 Gentoo Refleaks 3.x". Sadly, my commit 23401fb is not perfect, the test still fails when the system load is high: But since tests are re-run sequentially, I hope that it will be fine. I close the issue. I will reopen it if the bug reoccurs. |
We have some similar failures on the x86 Gentoo Refleaks 3.7 buildbots: http://buildbot.python.org/all/#/builders/114/builds/157/steps/4/logs/stdio It seems that is due to a high load on the buildbot but I am surprised that this is not mitigated after PR 8059. |
Ok ok, let me be honest with myself, my *workaround* change is not reliable :-) |
Even if the code isn't perfect, I didn't see the failure recently. So I close the bug again. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: