classification
Title: MainThread association logic is fragile
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: asvetlov, docs@python, eric.snow, ncoghlan, pitrou, tim.peters
Priority: normal Keywords:

Created on 2017-09-19 13:00 by pitrou, last changed 2017-09-20 08:47 by ncoghlan.

Files
File name Uploaded Description Edit
mainthread2.py pitrou, 2017-09-19 13:00
Messages (8)
msg302521 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-19 13:00
The threading main_thread() instance is associated to the thread that first imports the threading module.  In usual circumstances, this will indeed be the interpreter's main thread.  However, it is not always the case.  See attached reproducer.

$ ./python mainthread2.py 
child thread: <_MainThread(MainThread, started 140399567398656)>
main thread: <_DummyThread(Dummy-1, started daemon 140399588386560)>
Exception ignored in: <module 'threading' from '/home/antoine/cpython/default/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/antoine/cpython/default/Lib/threading.py", line 1268, in _shutdown
    assert tlock.locked()
AssertionError:
msg302523 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-19 13:07
Fixing this will require thinking out what the "main thread" of an interpreter really is.  Is it the thread in which PyInterpreterState_New() is called?

(note this is not the same thing as the "main thread" in the signal module, which is process-global)
msg302583 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2017-09-20 02:28
Is there a problem here?  I haven't heard of anyone even wondering about this before.  threading.py worries about Threads created _by_ threading.py, plus the magical (from its point of view) thread that first imports threading.py.  Users mix in `_thread` threads, or raw C threads from extension modules, essentially at their own risk.  Which risks are minimal, but can have visible consequences.

I don't view that as being a real problem.  It might help if, e.g., a Wiki somewhere spelled out the consequences under different Python implementations (while I don't know for sure, I _expect_ the current docs just say "in normal conditions, the main thread is the thread from which the Python interpreter was started" because it doesn't want to over-specify the behavior in an area nobody really cares about).
msg302589 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-09-20 04:24
One place where this came up recently is in working out precisely how a Python-level subinterpreter API will interact with the threading API: https://mail.python.org/pipermail/python-dev/2017-September/149566.html

That said, I do agree with Tim that the status quo isn't broken per se: we renamed `thread` to `_thread` in Python 3 for a reason, and that reason is that you really need to know how Python's threading internals work to do it safely.

However, I do think we can treat this as a documentation enhancement request, where the `_thread` module docs could point out some of the requirements to ensure that low-level thread manipulation plays nice with the threading module.
msg302601 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-20 08:26
> Is there a problem here?  I haven't heard of anyone even wondering about this before.

This came while working on the PEP 556 implementation.
msg302602 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-20 08:26
> That said, I do agree with Tim that the status quo isn't broken per se: we renamed `thread` to `_thread` in Python 3 for a reason

This is not caused by `_thread` specifically, but any background thread created by C code that might invoke Python code.  The reproducer just uses the `_thread` module out of convenience.
msg302603 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-20 08:31
I'm not sure this is an issue worth fixing, but I don't think it's a good idea to document such quirky semantics.
msg302606 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-09-20 08:47
We had the quirks of import related threading deadlocks documented for a long time, not as a promise, but as a debugging aid (and a recommendation for "don't do that").

I'd see this as being similar: we'd document that if you're using the _thread module, or otherwise allowing operating system threads not managed by the threading module to call in and run arbitrary Python code, then you should run "import threading" early in the actual main thread to make sure it gets associated correctly.
History
Date User Action Args
2017-09-20 08:47:19ncoghlansetmessages: + msg302606
2017-09-20 08:31:10pitrousetmessages: + msg302603
2017-09-20 08:26:57pitrousetmessages: + msg302602
2017-09-20 08:26:02pitrousetmessages: + msg302601
2017-09-20 04:24:35ncoghlansetnosy: + docs@python
messages: + msg302589

assignee: docs@python
components: + Documentation
type: behavior -> enhancement
2017-09-20 02:28:41tim.peterssetmessages: + msg302583
2017-09-19 13:07:05pitrousetnosy: + ncoghlan, eric.snow
messages: + msg302523
2017-09-19 13:00:23pitrousetnosy: + tim.peters, asvetlov
2017-09-19 13:00:11pitroucreate