classification
Title: multiprocess: catch explicitly SystemExit in the Server class
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: bquinlan, davin, pablogsal, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2021-04-16 14:41 by vstinner, last changed 2021-04-16 17:43 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25441 merged vstinner, 2021-04-16 14:47
Messages (3)
msg391205 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-16 14:41
In bpo-43843, I modified libregrtest to log a warning if a thread raises an uncaught exception. test_concurrent_futures now logs two warnings.

s390x Debian 3.x:
https://buildbot.python.org/all/#/builders/49/builds/1124

AMD64 RHEL7 3.x:
https://buildbot.python.org/all/#/builders/15/builds/1074

0:03:18 load avg: 4.78 [302/427] test_concurrent_futures passed (2 min 46 sec) -- running: test_tokenize (37.8 sec), test_peg_generator (59.2 sec)
Warning -- Uncaught thread exception: SystemExit
Warning -- Uncaught thread exception: SystemExit


The problem can be seen with test_ressources_gced_in_workers():

$ ./python -m test test_concurrent_futures -m test.test_concurrent_futures.ProcessPoolForkProcessPoolExecutorTest.test_ressources_gced_in_workers -v
(...)
test_ressources_gced_in_workers (test.test_concurrent_futures.ProcessPoolForkProcessPoolExecutorTest) ... 
Warning -- Uncaught thread exception: SystemExit
Warning -- Uncaught thread exception: SystemExit
(...)
Tests result: SUCCESS

I propose to explicitly catch this SystemExit. We can take the opportunity to call c.close() explicitly: see attached PR.
msg391207 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-16 14:48
Oh sorry, I was confused. While the issue is seen in test_concurrent_futures, my PR 25441 changes the multiprocessing module.
msg391240 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-16 17:42
New changeset 7c29ae1f0585378dba4d220a2c0fb5dd49fdab3e by Victor Stinner in branch 'master':
bpo-43867: multiprocessing Server catchs SystemExit (GH-25441)
https://github.com/python/cpython/commit/7c29ae1f0585378dba4d220a2c0fb5dd49fdab3e
History
Date User Action Args
2021-04-16 17:43:28vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-04-16 17:42:45vstinnersetmessages: + msg391240
2021-04-16 14:48:48vstinnersetmessages: + msg391207
2021-04-16 14:48:15vstinnersettitle: concurrent.futures: handle explicitly SystemExit in managers.Server -> multiprocess: catch explicitly SystemExit in the Server class
2021-04-16 14:47:42vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request24171
2021-04-16 14:41:31vstinnercreate