classification
Title: Daemon threads must be forbidden in subinterpreters
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, vstinner
Priority: normal Keywords: patch

Created on 2019-06-13 09:36 by vstinner, last changed 2019-07-04 16:30 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
subinterp_daemon_thread.py vstinner, 2019-06-13 09:36
Pull Requests
URL Status Linked Edit
PR 14049 merged vstinner, 2019-06-13 10:07
PR 14584 merged vstinner, 2019-07-04 10:23
Messages (4)
msg345485 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 09:36
Py_EndInterpreter() calls threading._shutdown() which waits for non-daemon threads spawned in the subinterpreters. Problem: daemon threads continue to run after threading._shutdown(), but they rely on an interpreter which is being finalized and then deleted.

Attached example shows the problem:

$ ./python subinterp_daemon_thread.py 
hello from daemon thread
Fatal Python error: Py_EndInterpreter: not the last thread

Current thread 0x00007f13e5926740 (most recent call first):
  File "subinterp_daemon_thread.py", line 23 in <module>
Aborted (core dumped)

Catching the bug in Py_EndInterpreter() is too late. IMHO we must simply deny daemon threads by design in subinterpreters for safety.

In the main interpreter, we provide best effort to prevent crash at exit, but IMHO the implementation is ugly :-( ceval.c uses exit_thread_if_finalizing(): it immediately exit the current daemon thread if the threads attempts to acquire or release the GIL, whereas the interpreter is gone. Problem: we cannot release/clear some data structure at Python exit because of that. So Py_Finalize() may leak some memory by design, because of daemon threads.

IMHO we can be way stricter in subinterpreters.

I suggest to only modify Python 3.9.
msg345612 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-14 16:55
New changeset 066e5b1a917ec2134e8997d2cadd815724314252 by Victor Stinner in branch 'master':
bpo-37266: Daemon threads are now denied in subinterpreters (GH-14049)
https://github.com/python/cpython/commit/066e5b1a917ec2134e8997d2cadd815724314252
msg345613 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-14 16:56
Daemon threads must die. That's a first step towards their death!
msg347287 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-04 16:30
New changeset b4e68960b90627422325fdb75f463df1e4153c6e by Victor Stinner in branch 'master':
bpo-37266: Add bpo number to the What's New entry (GH614584)
https://github.com/python/cpython/commit/b4e68960b90627422325fdb75f463df1e4153c6e
History
Date User Action Args
2019-07-04 16:30:42vstinnersetmessages: + msg347287
2019-07-04 10:23:34vstinnersetpull_requests: + pull_request14402
2019-06-14 16:56:51vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg345613

stage: patch review -> resolved
2019-06-14 16:55:27vstinnersetmessages: + msg345612
2019-06-13 10:07:07vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request13911
2019-06-13 10:05:26xtreaksetnosy: + eric.snow
2019-06-13 09:36:05vstinnercreate