This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing.Pool: emit ResourceWarning
Type: Stage: resolved
Components: Tests Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2018-12-06 01:00 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 10974 merged vstinner, 2018-12-06 01:01
PR 10986 merged vstinner, 2018-12-06 10:26
PR 10987 merged miss-islington, 2018-12-06 10:57
PR 10988 merged miss-islington, 2018-12-06 10:57
PR 11223 merged vstinner, 2018-12-18 21:44
PR 11227 merged miss-islington, 2018-12-18 22:54
Messages (10)
msg331201 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-06 01:00
Since 2 years, I'm fixing frequently "dangling thread" and "dangling process" warnings on buildbots. These bugs are really hard to reproduce. They usually require to get access to a specific buildbot, simulate a specific workload, and get the proper timing to get the warning.

I propose to emit a ResourceWarning in multiprocessing.Pool destructor if the pool has not been cleaned properly. I'm not sure in which cases a warning should be emitted.

Attached PR is a WIP implementation.
msg331220 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-06 10:26
While testing PR 10974, I found two tests which emits ResourceWarning. I wrote 10986 to fix them.
msg331223 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-06 10:56
New changeset 388c8c208d9d09bd28289c1e4776b947d4d0f0f0 by Victor Stinner in branch 'master':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
https://github.com/python/cpython/commit/388c8c208d9d09bd28289c1e4776b947d4d0f0f0
msg331230 - (view) Author: miss-islington (miss-islington) Date: 2018-12-06 11:20
New changeset b7c67c4d510a7a72a35983cc168dbb2ce796cb8c by Miss Islington (bot) in branch '3.7':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
https://github.com/python/cpython/commit/b7c67c4d510a7a72a35983cc168dbb2ce796cb8c
msg331231 - (view) Author: miss-islington (miss-islington) Date: 2018-12-06 11:23
New changeset e44b5b2afa6fe2966d8caff45e36c0980413bb86 by Miss Islington (bot) in branch '3.6':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
https://github.com/python/cpython/commit/e44b5b2afa6fe2966d8caff45e36c0980413bb86
msg331823 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-14 12:42
See this discussion:
[Python-Dev] Usage of the multiprocessing API and object lifetime
https://mail.python.org/pipermail/python-dev/2018-December/155946.html
msg332097 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-18 22:54
New changeset 6cdce3ddef805e11d75142f3e20e23c3fe21fdf4 by Victor Stinner in branch 'master':
bpo-35424: Fix test_multiprocessing_main_handling (GH-11223)
https://github.com/python/cpython/commit/6cdce3ddef805e11d75142f3e20e23c3fe21fdf4
msg332100 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-18 23:43
New changeset c74e7c48ba4f8437d4c8b402098b8cefc33a28a9 by Victor Stinner (Miss Islington (bot)) in branch '3.7':
bpo-35424: Fix test_multiprocessing_main_handling (GH-11223) (GH-11227)
https://github.com/python/cpython/commit/c74e7c48ba4f8437d4c8b402098b8cefc33a28a9
msg332249 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-20 19:33
New changeset 9a8d1d7562b11969034b92217fe66aab7a951fb6 by Victor Stinner in branch 'master':
bpo-35424: emit ResourceWarning at multiprocessing.Pool destruction (GH-10974)
https://github.com/python/cpython/commit/9a8d1d7562b11969034b92217fe66aab7a951fb6
msg332251 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-20 19:37
The first implementation of my PR also emitted ResourceWarning when .join() hasn't been called, but the merged PR only emit a warning if .close() and .terminated() have not been called: if the pool is still running.
History
Date User Action Args
2022-04-11 14:59:08adminsetgithub: 79605
2018-12-20 19:37:22vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg332251

stage: patch review -> resolved
2018-12-20 19:33:57vstinnersetmessages: + msg332249
2018-12-18 23:43:31vstinnersetmessages: + msg332100
2018-12-18 22:54:43miss-islingtonsetpull_requests: + pull_request10462
2018-12-18 22:54:35vstinnersetmessages: + msg332097
2018-12-18 21:44:01vstinnersetpull_requests: + pull_request10458
2018-12-14 12:42:01vstinnersetmessages: + msg331823
2018-12-06 11:23:22miss-islingtonsetmessages: + msg331231
2018-12-06 11:20:54miss-islingtonsetnosy: + miss-islington
messages: + msg331230
2018-12-06 10:57:16miss-islingtonsetpull_requests: + pull_request10229
2018-12-06 10:57:07miss-islingtonsetpull_requests: + pull_request10228
2018-12-06 10:56:54vstinnersetmessages: + msg331223
2018-12-06 10:26:45vstinnersetmessages: + msg331220
2018-12-06 10:26:09vstinnersetpull_requests: + pull_request10227
2018-12-06 01:01:46vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request10216
2018-12-06 01:00:34vstinnercreate