classification
Title: Add a way to determine if the current thread has the import lock
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.snow, gvanrossum, ncoghlan
Priority: normal Keywords:

Created on 2014-12-17 02:14 by gvanrossum, last changed 2014-12-17 23:45 by gvanrossum.

Messages (8)
msg232792 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2014-12-17 02:14
imp.lock_held() returns True if *any* thread is currently importing something. I'd like to have an API that can tell whether the *current* thread has the import lock.  The implementation keeps track of this, but doesn't make the info available.

Use case: print a warning/error if a network call is made during import.
msg232802 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-12-17 05:59
Keep in mind that the global import lock is only held long enough to create a module-level lock.  The cache of module locks is found at line 166 of Lib/importlib/_bootstrap.py and the code related to module-level locking follows.  So unfortunately it won't be as simple as just getting the thread info from the global import lock.

Regardless, would it be useful to expose a function in importlib.util that indicates whether or not the current thread is importing a module (actually the names of the modules it is importing)?  Something like that would probably check both the global import lock and the module level locks, but make it unnecessary to expose the locks directly.  Given the situation with module-level locks I image such a helper is going to be desirable.
msg232811 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-12-17 11:14
Perhaps an API like "importlib.util.locks_held()" that returns a list of module names?

We'd then just iterate over the _module_locks() dictionary, looking for locks where the owner matched the current thread id (alternatively, if speed was critical for Guido's use case, add a separate reverse mapping from tid to locks held)

(Unless I've missed something, we don't run user code with the global import lock held any more)
msg232819 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-12-17 14:36
> (Unless I've missed something, we don't run user code with the global
import lock held any more)

Ah.  You are correct.
msg232824 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2014-12-17 16:29
Oh. I'd forgotten the PY3 situation is completely different from the PY2
situation. We're still stuck here in PY2 land where there's just the one
import lock. I guess not even ctypes can help us find out whether the
current thread is holding the import lock (which is the condition we're
trying to test for so we can print the warning). Our current hack is to
check lock_held() and if it is held wait for up to 5 second (in 10 msec
increments) to see if it goes away -- if it doesn't go away it would seem
we are holding it in the current thread and then we fail with a rude
assert. If it does go away within that time we assume some other thread was
doing some lazy importing and we hope they'll be done soon because the
whole world is waiting for them.

On Wed, Dec 17, 2014 at 6:36 AM, Eric Snow <report@bugs.python.org> wrote:
>
>
> Eric Snow added the comment:
>
> > (Unless I've missed something, we don't run user code with the global
> import lock held any more)
>
> Ah.  You are correct.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23068>
> _______________________________________
>
msg232831 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-12-17 23:14
Looking at the implementation of PyImport_ImportModuleNoBlock, you should be able to invoke that via ctypes with a nonsense module name to probe for whether or not the current thread has the import lock.

A call like 'PyImport_ImportModuleNoBlock("this-is-not-a-legal-module-name")' will always fail with ImportError, but the exception message will be different if another thread holds the import lock. Specifically, it will end with "because the import lock is held by another thread".
msg232832 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-12-17 23:20
You'd still need to check lock_held() to see if *some* thread is holding the import lock. The non-blocking import API should then let you determine if that thread is the current one.
msg232837 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2014-12-17 23:45
Interesting. The Dropbox server team thanks you for the suggestion!
History
Date User Action Args
2014-12-17 23:45:12gvanrossumsetmessages: + msg232837
2014-12-17 23:20:25ncoghlansetmessages: + msg232832
2014-12-17 23:14:40ncoghlansetmessages: + msg232831
2014-12-17 16:29:35gvanrossumsetmessages: + msg232824
2014-12-17 14:36:48eric.snowsetmessages: + msg232819
2014-12-17 11:14:32ncoghlansetmessages: + msg232811
2014-12-17 05:59:41eric.snowsetnosy: + eric.snow, brett.cannon, ncoghlan
messages: + msg232802
2014-12-17 02:14:37gvanrossumcreate