classification
Title: test_multiprocessing_spawn and test_multiprocessing_forkserver leak dangling processes
Type: Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: haypo
Priority: normal Keywords:

Created on 2017-07-28 15:02 by haypo, last changed 2017-08-16 11:17 by haypo. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3103 merged haypo, 2017-08-16 10:28
PR 3104 merged haypo, 2017-08-16 10:46
PR 3105 merged haypo, 2017-08-16 10:48
Messages (11)
msg299392 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-28 15:02
It's just a warning, but later I would like to make a buildbot fail on such warnings, so I would like to first fix all these warnings.

http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.x/builds/59/steps/test/logs/stdio

0:46:17 [119/406] test_multiprocessing_spawn passed (2667 sec) -- running: test_asyncio (290 sec), test_userstring (54 sec), test_regrtest (471 sec)
beginning 6 repetitions
123456
...Warning -- Dangling processes: {<Process(Process-1235, stopped daemon)>}
...
msg300118 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 21:33
With the commit 957d0e9b59bd27ca7c473560634d8b5dbe66338c (bpo-26762), _test_multiprocessing now fails with ENV_CHANGED if a warning is emitted. It allowed to catch a warning on x86-64 El Capitan 3.x:

http://buildbot.python.org/all/builders/x86-64%20El%20Capitan%203.x/builds/581/steps/test/logs/stdio

test_enable_logging (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_boundaries (test.test_multiprocessing_spawn.WithProcessesTestPoll) ... ok
msg300122 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 21:43
Another warning on x86 Tiger 3.x:

http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/1043/steps/test/logs/stdio

test_rapid_restart (test.test_multiprocessing_forkserver.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok
msg300137 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 23:58
See also bpo-30846.
msg300334 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 08:45
I marked bpo-30846 as a duplicate of this issue. Copy of its message:

http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.6/builds/517/steps/test/logs/stdio

======================================================================
ERROR: test_rapid_restart (test.test_multiprocessing_fork.WithProcessesTestManagerRestart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/test/_test_multiprocessing.py", line 2410, in test_rapid_restart
    manager.start()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/managers.py", line 517, in start
    self._address = reader.recv()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.6.gps-debian-profile-opt.nondebug/build/Lib/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
(...)
Re-running test 'test_multiprocessing_fork' in verbose mode
(...)
test_rapid_restart (test.test_multiprocessing_fork.WithManagerTestManagerRestart) ... ok
msg300337 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 08:48
AppVeyor:

https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.5341

test_lock_context (test.test_multiprocessing_spawn.WithProcessesTestLock) ... ok
test_rlock (test.test_multiprocessing_spawn.WithProcessesTestLock) ... ok
test_enable_logging (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-174, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
msg300343 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 09:34
http://buildbot.python.org/all/builders/x86-64%20El%20Capitan%203.x/builds/590/steps/test/logs/stdio

test_enable_logging (test.test_multiprocessing_forkserver.WithProcessesTestLogging) ... ok
test_level (test.test_multiprocessing_forkserver.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_forkserver.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_forkserver.WithProcessesTestPicklingConnections) ... ok

(...)


test_level (test.test_multiprocessing_spawn.WithProcessesTestLogging) ... ok
test_rapid_restart (test.test_multiprocessing_spawn.WithProcessesTestManagerRestart) ... ok
Warning -- Dangling processes: {<Process(Process-181, stopped daemon)>}
test_access (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
test_pickling (test.test_multiprocessing_spawn.WithProcessesTestPicklingConnections) ... ok
msg300345 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 10:46
New changeset 17657bb9458ff8f8804b7637d61686a68f4b9471 by Victor Stinner in branch 'master':
bpo-31069, test_multiprocessing: Fix dangling process (#3103)
https://github.com/python/cpython/commit/17657bb9458ff8f8804b7637d61686a68f4b9471
msg300348 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 11:14
New changeset 78c66a641a0d4fe61356ebfae8772e4fae24dcb7 by Victor Stinner in branch '3.6':
bpo-31069, test_multiprocessing: Fix dangling process (#3103) (#3104)
https://github.com/python/cpython/commit/78c66a641a0d4fe61356ebfae8772e4fae24dcb7
msg300349 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 11:14
New changeset 883520a846d87f01a14636978cfa9cee1ecc07c1 by Victor Stinner in branch '2.7':
bpo-31069, test_multiprocessing: Fix dangling process (#3103) (#3105)
https://github.com/python/cpython/commit/883520a846d87f01a14636978cfa9cee1ecc07c1
msg300350 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-16 11:17
Ok, the main bug should have been fixed. I will reopen it if it's not the case, or open a new issue if a second bug is seen on buildbots.
History
Date User Action Args
2017-08-16 11:17:45hayposetstatus: open -> closed
resolution: fixed
messages: + msg300350

stage: resolved
2017-08-16 11:14:42hayposetmessages: + msg300349
2017-08-16 11:14:40hayposetmessages: + msg300348
2017-08-16 10:48:19hayposetpull_requests: + pull_request3144
2017-08-16 10:46:54hayposetpull_requests: + pull_request3143
2017-08-16 10:46:06hayposetmessages: + msg300345
2017-08-16 10:28:38hayposetpull_requests: + pull_request3142
2017-08-16 09:34:49hayposetmessages: + msg300343
2017-08-16 09:34:29hayposettitle: test_multiprocessing_spawn leaks a dangling process -> test_multiprocessing_spawn and test_multiprocessing_forkserver leak dangling processes
2017-08-16 08:48:56hayposetmessages: + msg300337
2017-08-16 08:46:19haypolinkissue30846 superseder
2017-08-16 08:45:58hayposetmessages: + msg300334
2017-08-10 23:58:17hayposetmessages: + msg300137
2017-08-10 23:57:39hayposettitle: test_multiprocessing_spawn leaked a dangling process -> test_multiprocessing_spawn leaks a dangling process
2017-08-10 21:43:54hayposetmessages: + msg300122
2017-08-10 21:33:45hayposettitle: test_multiprocessing_spawn leaked a process on AMD64 Windows8.1 Refleaks 3.x -> test_multiprocessing_spawn leaked a dangling process
2017-08-10 21:33:31hayposetmessages: + msg300118
2017-07-28 15:02:05haypocreate