classification
Title: Improving wording on the thread-safeness of import
Type: enhancement Stage:
Components: Documentation Versions:
process
Status: pending Resolution: out of date
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, iritkatriel, loewis, valhallasw
Priority: normal Keywords:

Created on 2012-06-17 14:23 by valhallasw, last changed 2020-12-31 12:42 by iritkatriel.

Files
File name Uploaded Description Edit
deadlock.py valhallasw, 2012-06-17 14:26 Example + explanation of an import deadlock
Messages (4)
msg163068 - (view) Author: Merlijn van Deen (valhallasw) * Date: 2012-06-17 14:23
http://docs.python.org/library/threading.html#importing-in-threaded-code

Currently, the documentation states
"Firstly, other than in the main module, an import should not have the side effect of spawning a new thread and then waiting for that thread in any way. Failing to abide by this restriction can lead to a deadlock if the spawned thread directly or indirectly attempts to import a module."

which, I think, fails to make the main point: a call to import acquires the import lock. A call to import from a second thread will thus block.

As such, I would suggest rephrasing it to something like:
"Firstly, an import acquires the import lock for that thread. Therefore, the import should not have the side effect of waiting for a different thread in any way, as this can lead to a deadlock if that thread directly or indirectly attempts to import a module."

There are two additional points that might be interesting to note:
(1) Any module can be imported. If the import causes a deadlock, that is a bad thing. Every module *will* be imported by tools such as nosetests.
(1b) so: never, ever, have code that causes locks in a different thread  in module level code witout 'if __name__=="__main__" ' blocks?

(2) The lock is also acquired if a module has already been imported. For instance, in

import sys # (1)
def f():
    import sys # (2)

the import lock is acquired in (1) /and/ (2).


Adding example code and/or a flow diagram might be a bit too much, but it does clarify how easy it is to make this mistake. See the attached for an example (both a simple example script, as well as a flow diagram explaining what happens).
msg163069 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-06-17 14:51
> which, I think, fails to make the main point:

I disagree. It currently makes it main point, but stops doing so
under your rephrasing. The main point of that section is

"While the import machinery is thread-safe, there are two key
restrictions on threaded imports due to inherent limitations in the way
that thread-safety is provided:"

The existence of an import lock is deliberately omitted from the text,
and the reader is supposed to abide by the restriction as written
regardless of the motivation behind it.

> Adding example code and/or a flow diagram might be a bit too much,
> but it does clarify how easy it is to make this mistake. See the
> attached for an example (both a simple example script, as well as a
> flow diagram explaining what happens).

The entire notion of an import lock is obsolete. Python 3.3 does not
have that anymore.
msg163071 - (view) Author: Merlijn van Deen (valhallasw) * Date: 2012-06-17 15:38
First off, thank you for your response.

> The existence of an import lock is deliberately omitted from the text,
> and the reader is supposed to abide by the restriction as written
> regardless of the motivation behind it.

> The entire notion of an import lock is obsolete. Python 3.3 does not
> have that anymore.

" This warning is still valid but for a different reason " or " this warning is no longer valid in 3.3 "?


Assuming the first (which is what I guess based on the fact the deadlock still occurs in 3.3), I think the text can still be improved; the current wording suggests to me

a) it's OK to wait for a thread as long as you did not create it

and

b) it's OK to import something that waits for a thread as long as you do it from the main module

 - while both cases can still lead to a deadlock. 

so, leaving the implementation details out, this is my suggestion:

"Firstly, an import should not have the side effect of waiting for a thread in any way. This can lead to a deadlock if that thread directly or indirectly attempts to import a module."
msg384113 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-12-31 12:42
The threading doc no longer mentions import at all. Any objections to closing this issue as out of date?  Or is there anything else to look into here?
History
Date User Action Args
2020-12-31 12:42:57iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg384113

resolution: out of date
2012-06-17 15:38:47valhallaswsetmessages: + msg163071
2012-06-17 14:51:23loewissetnosy: + loewis
messages: + msg163069
2012-06-17 14:26:08valhallaswsetfiles: + deadlock.py
2012-06-17 14:25:16valhallaswsetfiles: - deadlock.py
2012-06-17 14:23:55valhallaswcreate