classification
Title: Improving wording on the thread-safeness of import
Type: enhancement Stage:
Components: Documentation Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, loewis, valhallasw
Priority: normal Keywords:

Created on 2012-06-17 14:23 by valhallasw, last changed 2012-06-17 15:38 by valhallasw.

Files
File name Uploaded Description Edit
deadlock.py valhallasw, 2012-06-17 14:26 Example + explanation of an import deadlock
Messages (3)
msg163068 - (view) Author: Merlijn van Deen (valhallasw) * Date: 2012-06-17 14:23
http://docs.python.org/library/threading.html#importing-in-threaded-code

Currently, the documentation states
"Firstly, other than in the main module, an import should not have the side effect of spawning a new thread and then waiting for that thread in any way. Failing to abide by this restriction can lead to a deadlock if the spawned thread directly or indirectly attempts to import a module."

which, I think, fails to make the main point: a call to import acquires the import lock. A call to import from a second thread will thus block.

As such, I would suggest rephrasing it to something like:
"Firstly, an import acquires the import lock for that thread. Therefore, the import should not have the side effect of waiting for a different thread in any way, as this can lead to a deadlock if that thread directly or indirectly attempts to import a module."

There are two additional points that might be interesting to note:
(1) Any module can be imported. If the import causes a deadlock, that is a bad thing. Every module *will* be imported by tools such as nosetests.
(1b) so: never, ever, have code that causes locks in a different thread  in module level code witout 'if __name__=="__main__" ' blocks?

(2) The lock is also acquired if a module has already been imported. For instance, in

import sys # (1)
def f():
    import sys # (2)

the import lock is acquired in (1) /and/ (2).


Adding example code and/or a flow diagram might be a bit too much, but it does clarify how easy it is to make this mistake. See the attached for an example (both a simple example script, as well as a flow diagram explaining what happens).
msg163069 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-06-17 14:51
> which, I think, fails to make the main point:

I disagree. It currently makes it main point, but stops doing so
under your rephrasing. The main point of that section is

"While the import machinery is thread-safe, there are two key
restrictions on threaded imports due to inherent limitations in the way
that thread-safety is provided:"

The existence of an import lock is deliberately omitted from the text,
and the reader is supposed to abide by the restriction as written
regardless of the motivation behind it.

> Adding example code and/or a flow diagram might be a bit too much,
> but it does clarify how easy it is to make this mistake. See the
> attached for an example (both a simple example script, as well as a
> flow diagram explaining what happens).

The entire notion of an import lock is obsolete. Python 3.3 does not
have that anymore.
msg163071 - (view) Author: Merlijn van Deen (valhallasw) * Date: 2012-06-17 15:38
First off, thank you for your response.

> The existence of an import lock is deliberately omitted from the text,
> and the reader is supposed to abide by the restriction as written
> regardless of the motivation behind it.

> The entire notion of an import lock is obsolete. Python 3.3 does not
> have that anymore.

" This warning is still valid but for a different reason " or " this warning is no longer valid in 3.3 "?


Assuming the first (which is what I guess based on the fact the deadlock still occurs in 3.3), I think the text can still be improved; the current wording suggests to me

a) it's OK to wait for a thread as long as you did not create it

and

b) it's OK to import something that waits for a thread as long as you do it from the main module

 - while both cases can still lead to a deadlock. 

so, leaving the implementation details out, this is my suggestion:

"Firstly, an import should not have the side effect of waiting for a thread in any way. This can lead to a deadlock if that thread directly or indirectly attempts to import a module."
History
Date User Action Args
2012-06-17 15:38:47valhallaswsetmessages: + msg163071
2012-06-17 14:51:23loewissetnosy: + loewis
messages: + msg163069
2012-06-17 14:26:08valhallaswsetfiles: + deadlock.py
2012-06-17 14:25:16valhallaswsetfiles: - deadlock.py
2012-06-17 14:23:55valhallaswcreate