classification
Title: multiprocessing.Pool: emit ResourceWarning
Type: Stage: patch review
Components: Tests Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2018-12-06 01:00 by vstinner, last changed 2018-12-14 12:42 by vstinner.

Pull Requests
URL Status Linked Edit
PR 10974 open vstinner, 2018-12-06 01:01
PR 10986 merged vstinner, 2018-12-06 10:26
PR 10987 merged miss-islington, 2018-12-06 10:57
PR 10988 merged miss-islington, 2018-12-06 10:57
Messages (6)
msg331201 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-06 01:00
Since 2 years, I'm fixing frequently "dangling thread" and "dangling process" warnings on buildbots. These bugs are really hard to reproduce. They usually require to get access to a specific buildbot, simulate a specific workload, and get the proper timing to get the warning.

I propose to emit a ResourceWarning in multiprocessing.Pool destructor if the pool has not been cleaned properly. I'm not sure in which cases a warning should be emitted.

Attached PR is a WIP implementation.
msg331220 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-06 10:26
While testing PR 10974, I found two tests which emits ResourceWarning. I wrote 10986 to fix them.
msg331223 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-06 10:56
New changeset 388c8c208d9d09bd28289c1e4776b947d4d0f0f0 by Victor Stinner in branch 'master':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
https://github.com/python/cpython/commit/388c8c208d9d09bd28289c1e4776b947d4d0f0f0
msg331230 - (view) Author: miss-islington (miss-islington) Date: 2018-12-06 11:20
New changeset b7c67c4d510a7a72a35983cc168dbb2ce796cb8c by Miss Islington (bot) in branch '3.7':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
https://github.com/python/cpython/commit/b7c67c4d510a7a72a35983cc168dbb2ce796cb8c
msg331231 - (view) Author: miss-islington (miss-islington) Date: 2018-12-06 11:23
New changeset e44b5b2afa6fe2966d8caff45e36c0980413bb86 by Miss Islington (bot) in branch '3.6':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
https://github.com/python/cpython/commit/e44b5b2afa6fe2966d8caff45e36c0980413bb86
msg331823 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-14 12:42
See this discussion:
[Python-Dev] Usage of the multiprocessing API and object lifetime
https://mail.python.org/pipermail/python-dev/2018-December/155946.html
History
Date User Action Args
2018-12-14 12:42:01vstinnersetmessages: + msg331823
2018-12-06 11:23:22miss-islingtonsetmessages: + msg331231
2018-12-06 11:20:54miss-islingtonsetnosy: + miss-islington
messages: + msg331230
2018-12-06 10:57:16miss-islingtonsetpull_requests: + pull_request10229
2018-12-06 10:57:07miss-islingtonsetpull_requests: + pull_request10228
2018-12-06 10:56:54vstinnersetmessages: + msg331223
2018-12-06 10:26:45vstinnersetmessages: + msg331220
2018-12-06 10:26:09vstinnersetpull_requests: + pull_request10227
2018-12-06 01:01:46vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request10216
2018-12-06 01:00:34vstinnercreate