This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: MainThread association logic is fragile
Type: enhancement Stage: needs patch
Components: Documentation, Subinterpreters Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: aldwinaldwin, asvetlov, docs@python, eric.snow, fabioz, int19h, ncoghlan, pitrou, tim.peters, vstinner
Priority: normal Keywords:

Created on 2017-09-19 13:00 by pitrou, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
mainthread2.py pitrou, 2017-09-19 13:00
Messages (11)
msg302521 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-19 13:00
The threading main_thread() instance is associated to the thread that first imports the threading module.  In usual circumstances, this will indeed be the interpreter's main thread.  However, it is not always the case.  See attached reproducer.

$ ./python mainthread2.py 
child thread: <_MainThread(MainThread, started 140399567398656)>
main thread: <_DummyThread(Dummy-1, started daemon 140399588386560)>
Exception ignored in: <module 'threading' from '/home/antoine/cpython/default/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/antoine/cpython/default/Lib/threading.py", line 1268, in _shutdown
    assert tlock.locked()
AssertionError:
msg302523 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-19 13:07
Fixing this will require thinking out what the "main thread" of an interpreter really is.  Is it the thread in which PyInterpreterState_New() is called?

(note this is not the same thing as the "main thread" in the signal module, which is process-global)
msg302583 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2017-09-20 02:28
Is there a problem here?  I haven't heard of anyone even wondering about this before.  threading.py worries about Threads created _by_ threading.py, plus the magical (from its point of view) thread that first imports threading.py.  Users mix in `_thread` threads, or raw C threads from extension modules, essentially at their own risk.  Which risks are minimal, but can have visible consequences.

I don't view that as being a real problem.  It might help if, e.g., a Wiki somewhere spelled out the consequences under different Python implementations (while I don't know for sure, I _expect_ the current docs just say "in normal conditions, the main thread is the thread from which the Python interpreter was started" because it doesn't want to over-specify the behavior in an area nobody really cares about).
msg302589 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-09-20 04:24
One place where this came up recently is in working out precisely how a Python-level subinterpreter API will interact with the threading API: https://mail.python.org/pipermail/python-dev/2017-September/149566.html

That said, I do agree with Tim that the status quo isn't broken per se: we renamed `thread` to `_thread` in Python 3 for a reason, and that reason is that you really need to know how Python's threading internals work to do it safely.

However, I do think we can treat this as a documentation enhancement request, where the `_thread` module docs could point out some of the requirements to ensure that low-level thread manipulation plays nice with the threading module.
msg302601 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-20 08:26
> Is there a problem here?  I haven't heard of anyone even wondering about this before.

This came while working on the PEP 556 implementation.
msg302602 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-20 08:26
> That said, I do agree with Tim that the status quo isn't broken per se: we renamed `thread` to `_thread` in Python 3 for a reason

This is not caused by `_thread` specifically, but any background thread created by C code that might invoke Python code.  The reproducer just uses the `_thread` module out of convenience.
msg302603 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-20 08:31
I'm not sure this is an issue worth fixing, but I don't think it's a good idea to document such quirky semantics.
msg302606 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-09-20 08:47
We had the quirks of import related threading deadlocks documented for a long time, not as a promise, but as a debugging aid (and a recommendation for "don't do that").

I'd see this as being similar: we'd document that if you're using the _thread module, or otherwise allowing operating system threads not managed by the threading module to call in and run arbitrary Python code, then you should run "import threading" early in the actual main thread to make sure it gets associated correctly.
msg347264 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-04 10:19
Python internals already know who is the "main" thread: _PyRuntime.main_thread. It's maintained up to date, even after a fork, PyOS_AfterFork_Child() calls _PyRuntimeState_ReInitThreads() which does:

    // This was initially set in _PyRuntimeState_Init().
    runtime->main_thread = PyThread_get_thread_ident();

I already added _thread._is_main_interpreter() to deny spawning daemon threads in subinterpreters: bpo-37266.

We can add _thread._is_main_thread() which can reuse Modules/signalmodule.c code:

static int
is_main(_PyRuntimeState *runtime)
{
    unsigned long thread = PyThread_get_thread_ident();
    PyInterpreterState *interp = _PyRuntimeState_GetThreadState(runtime)->interp;
    return (thread == runtime->main_thread
            && interp == runtime->interpreters.main);
}

For example, this function is used by signal.signal:

    if (!is_main(runtime)) {
        PyErr_SetString(PyExc_ValueError,
                        "signal only works in main thread");
        return NULL;
    }
msg347266 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-04 10:31
bpo-37416 has been marked as a duplicate of this issue. It contains snippet.py to reproduce a bug.
msg358744 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-12-20 23:12
probably a duplicate: issue #39042 "Use the runtime's main thread ID in the threading module."
History
Date User Action Args
2022-04-11 14:58:52adminsetgithub: 75698
2020-06-03 16:42:33vstinnersetcomponents: + Subinterpreters
2019-12-20 23:12:47eric.snowsetmessages: + msg358744
2019-07-04 10:31:10vstinnersetmessages: + msg347266
2019-07-04 10:19:46vstinnersetmessages: + msg347264
2019-07-03 21:42:03pitrousetversions: + Python 3.8, Python 3.9, - Python 3.6, Python 3.7
2019-07-03 21:41:56pitrousetnosy: + vstinner, fabioz, aldwinaldwin, int19h
2019-07-03 21:41:08pitroulinkissue37416 superseder
2017-09-20 08:47:19ncoghlansetmessages: + msg302606
2017-09-20 08:31:10pitrousetmessages: + msg302603
2017-09-20 08:26:57pitrousetmessages: + msg302602
2017-09-20 08:26:02pitrousetmessages: + msg302601
2017-09-20 04:24:35ncoghlansetnosy: + docs@python
messages: + msg302589

assignee: docs@python
components: + Documentation
type: behavior -> enhancement
2017-09-20 02:28:41tim.peterssetmessages: + msg302583
2017-09-19 13:07:05pitrousetnosy: + ncoghlan, eric.snow
messages: + msg302523
2017-09-19 13:00:23pitrousetnosy: + tim.peters, asvetlov
2017-09-19 13:00:11pitroucreate