classification
Title: test_multiprocessing_spawn and test_multiprocessing_forkserver leak dangling processes
Type: Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: haypo
Priority: normal Keywords:

Created on 2017-07-28 15:02 by haypo, last changed 2017-09-15 16:40 by haypo. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3103 merged haypo, 2017-08-16 10:28
PR 3104 merged haypo, 2017-08-16 10:46
PR 3105 merged haypo, 2017-08-16 10:48
Messages (16)
msg299392 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-28 15:02
It's just a warning, but later I would like to make a buildbot fail on such warnings, so I would like to first fix all these warnings.

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.x/builds/59/steps/test/logs/stdio

0:46:17 [119/406] test_multiprocessing_spawn passed (2667 sec) -- running: test_asyncio (290 sec), test_userstring (54 sec), test_regrtest (471 sec)
beginning 6 repetitions
123456
...Warning -- Dangling processes: {<Process(Process-1235, stopped daemon)>}
...
msg300118 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 21:33
With the commit 957d0e9b59bd27ca7c473560634d8b5dbe66338c (bpo-26762), _test_multiprocessing now fails with ENV_CHANGED if a warning is emitted. It allowed to catch a warning on x86-64 El Capitan 3.x:

http://buildbot.python.org/all/builders/x86-64%20El%20Capitan%203.x/builds/581/steps/test/logs/stdio

test_enable_logging (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_boundaries (test.test_multiprocessing_spawn.WithProcessesTestPoll) ... ok
msg300122 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 21:43
Another warning on x86 Tiger 3.x:

http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/1043/steps/test/logs/stdio

test_rapid_restart (test.test_multiprocessing_forkserver.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok
msg300137 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 23:58
See also bpo-30846.
msg300334 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 08:45
I marked bpo-30846 as a duplicate of this issue. Copy of its message:

http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.6/builds/517/steps/test/logs/stdio

======================================================================
ERROR: test_rapid_restart (test.test_multiprocessing_fork.WithProcessesTestManagerRestart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/test/_test_multiprocessing.py", line 2410, in test_rapid_restart
    manager.start()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/managers.py", line 517, in start
    self._address = reader.recv()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
(...)
Re-running test 'test_multiprocessing_fork' in verbose mode
(...)
test_rapid_restart (test.test_multiprocessing_fork.WithManagerTestManagerRestart) ... ok
msg300337 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 08:48
AppVeyor:

https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.5341

test_lock_context (test.test_multiprocessing_spawn.WithProcessesTestLock) ... ok
test_rlock (test.test_multiprocessing_spawn.WithProcessesTestLock) ... ok
test_enable_logging (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-174, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
msg300343 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 09:34
http://buildbot.python.org/all/builders/x86-64%20El%20Capitan%203.x/builds/590/steps/test/logs/stdio

test_enable_logging (test.test_multiprocessing_forkserver.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_forkserver.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_forkserver.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok

(...)


test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
msg300345 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 10:46
New changeset 17657bb9458ff8f8804b7637d61686a68f4b9471 by Victor Stinner in branch 'master':
bpo-31069, test_multiprocessing: Fix dangling process (#3103)
https://github.com/python/cpython/commit/17657bb9458ff8f8804b7637d61686a68f4b9471
msg300348 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 11:14
New changeset 78c66a641a0d4fe61356ebfae8772e4fae24dcb7 by Victor Stinner in branch '3.6':
bpo-31069, test_multiprocessing: Fix dangling process (#3103) (#3104)
https://github.com/python/cpython/commit/78c66a641a0d4fe61356ebfae8772e4fae24dcb7
msg300349 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 11:14
New changeset 883520a846d87f01a14636978cfa9cee1ecc07c1 by Victor Stinner in branch '2.7':
bpo-31069, test_multiprocessing: Fix dangling process (#3103) (#3105)
https://github.com/python/cpython/commit/883520a846d87f01a14636978cfa9cee1ecc07c1
msg300350 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 11:17
Ok, the main bug should have been fixed. I will reopen it if it's not the case, or open a new issue if a second bug is seen on buildbots.
msg301339 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-05 17:13
Too bad, the bug is not dead, I reopen the issue.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/800/steps/test/logs/stdio

test_rlock (test.test_multiprocessing_forkserver.WithThreadsTestLock) ... ok
test_rapid_restart (test.test_multiprocessing_forkserver.WithThreadsTestManagerRestart) ... ok
Warning -- Dangling processes: {<ForkServerProcess(QueueManager-465, stopped)>}
test_boundaries (test.test_multiprocessing_forkserver.WithThreadsTestPoll) ... ok
test_dont_merge (test.test_multiprocessing_forkserver.WithThreadsTestPoll) ... ok
msg301969 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-12 16:03
Failure on AMD64 FreeBSD 10.x Shared 3.x:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/868/steps/test/logs/stdio

...
test_map_chunksize (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_map_handle_iterable_exception (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_map_no_failfast (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_map_unplicklable (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... skipped 'test not appropriate for threads'
test_release_task_refs (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_starmap (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_starmap_async (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_terminate (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_traceback (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
test_wrapped_exception (test.test_multiprocessing_forkserver.WithThreadsTestPool) ... ok
Warning -- Dangling threads: {<DummyProcess(Thread-158, started daemon 34461984768)>}
test_active_children (test.test_multiprocessing_forkserver.WithThreadsTestProcess) ... ok
msg301977 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-12 17:07
On the FreeBSD 10 buildbot, I ran test_multiprocessing_forkserver during 10 minutes twice in two terminals, while running "./python -m test -j2 -r -F" twice in two other terminals to stress the machine. I failed to reproduce the "Warning -- Dangling threads: {<DummyProcess(Thread-158, started daemon 34461984768)>}" warning.
msg302061 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-13 12:21
New test_multiprocessing_fork failure on AMD64 FreeBSD CURRENT Debug 3.x:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/869/steps/test/logs/stdio

(...)
test_traceback (test.test_multiprocessing_fork.WithThreadsTestPool) ... ok
test_wrapped_exception (test.test_multiprocessing_fork.WithThreadsTestPool) ... ok
Warning -- Dangling threads: {<DummyProcess(Thread-158, started daemon 34496780800)>}
test_active_children (test.test_multiprocessing_fork.WithThreadsTestProcess) ... ok
(...)
msg302273 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-15 16:40
I continued the effort to fix all "dangling threads/processes" in bpo-31234, so I close this issue.
History
Date User Action Args
2017-09-15 16:40:24hayposetstatus: open -> closed
resolution: fixed
messages: + msg302273
2017-09-13 12:21:16hayposetmessages: + msg302061
2017-09-12 17:07:00hayposetmessages: + msg301977
2017-09-12 16:03:04hayposetmessages: + msg301969
2017-09-05 17:13:26hayposetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg301339
2017-08-16 11:17:45hayposetstatus: open -> closed
resolution: fixed
messages: + msg300350

stage: resolved
2017-08-16 11:14:42hayposetmessages: + msg300349
2017-08-16 11:14:40hayposetmessages: + msg300348
2017-08-16 10:48:19hayposetpull_requests: + pull_request3144
2017-08-16 10:46:54hayposetpull_requests: + pull_request3143
2017-08-16 10:46:06hayposetmessages: + msg300345
2017-08-16 10:28:38hayposetpull_requests: + pull_request3142
2017-08-16 09:34:49hayposetmessages: + msg300343
2017-08-16 09:34:29hayposettitle: test_multiprocessing_spawn leaks a dangling process -> test_multiprocessing_spawn and test_multiprocessing_forkserver leak dangling processes
2017-08-16 08:48:56hayposetmessages: + msg300337
2017-08-16 08:46:19haypolinkissue30846 superseder
2017-08-16 08:45:58hayposetmessages: + msg300334
2017-08-10 23:58:17hayposetmessages: + msg300137
2017-08-10 23:57:39hayposettitle: test_multiprocessing_spawn leaked a dangling process -> test_multiprocessing_spawn leaks a dangling process
2017-08-10 21:43:54hayposetmessages: + msg300122
2017-08-10 21:33:45hayposettitle: test_multiprocessing_spawn leaked a process on AMD64 Windows8.1 Refleaks 3.x -> test_multiprocessing_spawn leaked a dangling process
2017-08-10 21:33:31hayposetmessages: + msg300118
2017-07-28 15:02:05haypocreate