http://buildbot.python.org/all/#/builders/132/builds/154

test_multiprocessing_spawn leaked [1, 2, 1] memory blocks, sum=4

When running "python -m test -R 2:3 test_multiprocessing_forkserver" on Windows, I saw some warnings about dangling threads. It may explain this issue.

bpo-33853 has been marked as a duplicate this bug.

> http://buildbot.python.org/all/#/builders/132/builds/154

That's the AMD64 Windows8.1 Refleaks 3.7 buildbot.

> bpo-33853 has been marked as a duplicate this bug.

Copy of Pablo's message:

The test `test_multiprocessing_spawn` is leaking memory according to the x86 Gentoo Refleaks 3.x buildbot:


x86 Gentoo Refleaks 3.x
http://buildbot.python.org/all/#/builders/1/builds/253

test_multiprocessing_spawn leaked [1, 2, 1] memory blocks, sum=4
1 test failed again:
    test_multiprocessing_spawn


x86 Gentoo Refleaks 3.7
http://buildbot.python.org/all/#/builders/114/builds/135

I just created bpo-33929: "test_multiprocessing_spawn: WithProcessesTestProcess.test_many_processes() leaks 5 handles on Windows". See also my PR 7827.

I also created bpo-33966: "test_multiprocessing_spawn.WithProcessesTestPool.test_traceback() leaks 4 handles on Windows".

I reproduced the issue on the Gentoo Refleak buildbot and I succeeded to bisect up to a single test: test_imap_unordered().

pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3
Run tests sequentially
0:00:00 load avg: 1.45 [1/1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
test_multiprocessing_spawn leaked [3, 2, 1] memory blocks, sum=6
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 6 sec 548 ms
Tests result: FAILURE

It doesn't look like a real leak, but more a cache which takes multiple iterations to be fully filled.

pydev@stormageddon ~/cpython $ ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 1:30
WARNING: Running tests with --huntrleaks/-R and less than 3 warmup repetitions can give false positives!
Run tests sequentially
0:00:00 load avg: 0.88 [1/1] test_multiprocessing_spawn
beginning 31 repetitions
1234567890123456789012345678901
...............................
test_multiprocessing_spawn leaked [4, 5, 1, 5, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=18
test_multiprocessing_spawn failed in 42 sec 470 ms

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 42 sec 490 ms
Tests result: FAILURE

> test_multiprocessing_spawn leaked [4, 5, 1, 5, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] memory blocks, sum=18

Sorry, I forgot to mention that I modified libregrtest to get this output:

diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py
index 6724488fcf..a3c50e21e0 100644
--- a/Lib/test/libregrtest/refleak.py
+++ b/Lib/test/libregrtest/refleak.py
@@ -101,7 +101,7 @@ def dash_R(the_module, test, indirect_test, huntrleaks):
     failed = False
     for deltas, item_name, checker in [
         (rc_deltas, 'references', check_rc_deltas),
-        (alloc_deltas, 'memory blocks', check_rc_deltas),
+        (alloc_deltas, 'memory blocks', check_fd_deltas),
         (fd_deltas, 'file descriptors', check_fd_deltas)
     ]:
         # ignore warmup runs

It failed for the first time on Ubuntu and then was successful for all the rest of 5-6 runs. I don't know why for the failure run it has load avg as 0.00 and how to get to this stage.

# Shell session

➜  cpython git:(master) uname -a
Linux ubuntu-s-1vcpu-1gb-blr1-01 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
➜  cpython git:(master) ./python
Python 3.8.0a0 (heads/master:d824ca7, Jul  3 2018, 06:50:05)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
➜  cpython git:(master) ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3
Run tests sequentially
0:00:00 load avg: 0.00 [1/1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
test_multiprocessing_spawn leaked [2, 2, 1] memory blocks, sum=5
test_multiprocessing_spawn failed

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 9 sec 221 ms
Tests result: FAILURE
➜  cpython git:(master) ./python -m test test_multiprocessing_spawn -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_imap_unordered -R 3:3
Run tests sequentially
0:00:00 load avg: 0.34 [1/1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 8 sec 822 ms
Tests result: SUCCESS

> It failed for the first time on Ubuntu and then was successful for all the rest of 5-6 runs.

The bug is random. But the problem is that sometimes, it fails. It must never fail, otherwise the buildbot fails randomly. The Gentoo Refleak buildbot runs multiple tests in parallel and so its system load is high, tests are run slower, making the failure more likely.

Anyway, I have a fix! PR 8059.

New changeset 23401fb960bb94e6ea62d2999527968d53d3fc65 by Victor Stinner in branch 'master':
bpo-33735: Fix test_multiprocessing random failure (GH-8059)
https://github.com/python/cpython/commit/23401fb960bb94e6ea62d2999527968d53d3fc65

New changeset 42b2f84a85fe3334c5d8098cf76dfa727b348c4f by Miss Islington (bot) in branch '3.7':
bpo-33735: Fix test_multiprocessing random failure (GH-8059)
https://github.com/python/cpython/commit/42b2f84a85fe3334c5d8098cf76dfa727b348c4f

New changeset 53fafafa340e292a1c9e4aacf5860999613c2b36 by Victor Stinner in branch '2.7':
bpo-33735: Fix test_multiprocessing random failure (GH-8059) (GH-8061)
https://github.com/python/cpython/commit/53fafafa340e292a1c9e4aacf5860999613c2b36

New changeset 3bd9d3b934eb8b27553f19a7ab1c97f95745c68f by Miss Islington (bot) in branch '3.6':
bpo-33735: Fix test_multiprocessing random failure (GH-8059)
https://github.com/python/cpython/commit/3bd9d3b934eb8b27553f19a7ab1c97f95745c68f

bpo-33984 has been marked as a duplicate of this issue: "test_multiprocessing_forkserver leaked [1, 2, 1] memory blocks on x86 Gentoo Refleaks 3.x".

Sadly, my commit 23401fb960bb94e6ea62d2999527968d53d3fc65 is not perfect, the test still fails when the system load is high:
https://bugs.python.org/issue33984#msg320967

But since tests are re-run sequentially, I hope that it will be fine.

I close the issue. I will reopen it if the bug reoccurs.

We have some similar failures on the x86 Gentoo Refleaks 3.7 buildbots:

http://buildbot.python.org/all/#/builders/114/builds/157/steps/4/logs/stdio
http://buildbot.python.org/all/#/builders/114/builds/155/steps/4/logs/stdio
----------------------------------------------------------------------
Ran 310 tests in 249.377s
OK (skipped=30)
.
test_multiprocessing_spawn leaked [1, 2, 2] memory blocks, sum=5
1 test failed again:
    test_multiprocessing_spawn
== Tests result: FAILURE then FAILURE ==

It seems that is due to a high load on the buildbot but I am surprised that this is not mitigated after PR 8059.

Ok ok, let me be honest with myself, my *workaround* change is not reliable :-)

Note for myself: the commit 127bd9bfd591c8ec1a97eb7f4037c8b884eef973, fix for bpo-34042 has no impact on this issue since this issue is about memory blocks and not references.

Even if the code isn't perfect, I didn't see the failure recently. So I close the bug again.