classification
Title: multiprocessing.Pool should join "dead" processes
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords:

Created on 2017-07-24 17:36 by vstinner, last changed 2017-07-26 15:55 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2849 closed vstinner, 2017-07-24 17:47
PR 2875 merged vstinner, 2017-07-25 23:57
PR 2879 merged vstinner, 2017-07-26 02:27
PR 2882 merged vstinner, 2017-07-26 02:51
Messages (6)
msg299002 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-24 17:36
With debug patches for bpo-26762, I noticed that some unit tests of test_multiprocessing_spawn leaks "dangling" processes:
---
haypo@selma$ ./python -m test --fail-env-changed test_multiprocessing_spawn -v --match test.test_multiprocessing_spawn.WithProcessesTestPool.test_context
== CPython 3.7.0a0 (heads/master:b364d9f, Jul 24 2017, 11:06:33) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
== Linux-4.11.9-200.fc25.x86_64-x86_64-with-fedora-25-Twenty_Five little-endian
== hash algorithm: siphash24 64bit
== cwd: /home/haypo/prog/python/master/build/test_python_20982
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 load avg: 0.16 [1/1] test_multiprocessing_spawn
test_context (test.test_multiprocessing_spawn.WithProcessesTestPool) ... ok
Warning -- Dangling processes: {<SpawnProcess(SpawnPoolWorker-6, stopped[SIGTERM] daemon)>}
Dangling processes: {<SpawnProcess(SpawnPoolWorker-6, stopped[SIGTERM] daemon)>}

----------------------------------------------------------------------
Ran 1 test in 1.342s

OK
test_multiprocessing_spawn failed (env changed)

1 test altered the execution environment:
    test_multiprocessing_spawn

Total duration: 1 sec
Tests result: ENV CHANGED
---

multiprocessing.Pool.terminate() doesn't call the join() method of a Process object if its is_alive() method returns false. But in practice, avoid the join() creates "dangling" processes.

Attached pull request fixes the warning: Pool.terminate() now calls join() on all processes including "dead" processes.
msg299170 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-25 23:59
Hum, I don't think that https://github.com/python/cpython/pull/2849 fixes the root issue.

The root issue is that calling Process.is_alive() doesn't update the _children list if the process completes. I wrote a fix for that:
https://github.com/python/cpython/pull/2875
msg299177 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 00:32
New changeset 2db64823c20538a6cfc6033661fab5711d2d4585 by Victor Stinner in branch 'master':
bpo-31019: Fix multiprocessing.Process.is_alive() (#2875)
https://github.com/python/cpython/commit/2db64823c20538a6cfc6033661fab5711d2d4585
msg299189 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 02:48
New changeset d0adfb25c5082046133a18fd185375508c1c334f by Victor Stinner in branch '3.6':
[3.6] bpo-26762, bpo-31019: Backport multiprocessing fixes from master to 3.6 (#2879)
https://github.com/python/cpython/commit/d0adfb25c5082046133a18fd185375508c1c334f
msg299245 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 15:54
New changeset b65cb8a35641675d44af84c9b18c40c6094f03ef by Victor Stinner in branch '2.7':
bpo-31019: Fix multiprocessing.Process.is_alive() (#2875) (#2882)
https://github.com/python/cpython/commit/b65cb8a35641675d44af84c9b18c40c6094f03ef
msg299246 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 15:55
I fixed multiprocessing.Process.is_alive() in 2.7, 3.6 and master. I close the issue.
History
Date User Action Args
2017-07-26 15:55:41vstinnersetstatus: open -> closed
versions: + Python 2.7, Python 3.6
messages: + msg299246

resolution: fixed
stage: resolved
2017-07-26 15:54:47vstinnersetmessages: + msg299245
2017-07-26 02:51:20vstinnersetpull_requests: + pull_request2934
2017-07-26 02:48:58vstinnersetmessages: + msg299189
2017-07-26 02:27:15vstinnersetpull_requests: + pull_request2931
2017-07-26 00:32:45vstinnersetmessages: + msg299177
2017-07-25 23:59:03vstinnersetmessages: + msg299170
2017-07-25 23:57:59vstinnersetpull_requests: + pull_request2926
2017-07-24 17:47:04vstinnersetpull_requests: + pull_request2898
2017-07-24 17:36:38vstinnercreate