Author josh.r
Recipients Denny Weinberg, Roman.Evstifeev, ammar2, belopolsky, josh.r, ncoghlan, palm.kevin
Date 2016-09-15.19:16:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
Nick: Looks like it's quite a bit more work than just a dict lookup. That PyImport_ImportModuleNoBlock call (which seems odd; the implementation of NoBlock is just to wrap the blocking function; guess we don't allow non-blocking imports anymore and this is just to avoid changing all the names elsewhere?) involves a *lot* more work than just a dict lookup (it devolves to a PyImport_Import call , which basically does everything involved in the import process aside from actually reading/parsing the file unconditionally, because of how weird __import__ overrides can be, I guess).

While it's not a perfect comparison, compare:

>>> import _strptime  # It's now cached

# Cache globals dict for fair comparison without globals() call overhead
>>> g = globals()     

# Reimport (this might be *more* expensive at C layer, see notes below)
>>> %timeit -r5 import _strptime
1000000 loops, best of 5: 351 ns per loop

# Dict lookup (should be at least a bit cheaper at C layer if done equivalently, using GetAttrId to avoid temporary str)
>>> %timeit -r5 g['_strptime']
10000000 loops, best of 5: 33.1 ns per loop

# Cached reference (should be *much* cheaper at C layer)
>>> %timeit -r5 _strptime
100000000 loops, best of 5: 19.1 ns per loop

Note: I'm a little unclear on whether a Python function implemented in C has its own globals, or whether it's simulated as part of the C module initialization); if it lacks globals, then the work done for PyImport_Import looks like it roughly doubles (it has to do all sorts of work to simulate globals and the like), so that 351 ns per re-import might actually be costlier in C.

Either way, it's a >10x increase in cost to reimport compared to a dict lookup, and ~18x speedup over using a cached reference (and like I said, I think the real cost of the cheaper options would be much less in C, so the multiplier is higher). Admittedly, in tests, empty string calls to `_strptime._strptime` take around 7.4 microseconds (with realistic calls taking 8.5-13.5 microseconds), so caching is saving maybe a third of a microsecond overhead, maybe 2.5%-4.5% of the work involved in the strptime call.
Date User Action Args
2016-09-15 19:16:43josh.rsetrecipients: + josh.r, ncoghlan, belopolsky, palm.kevin, Roman.Evstifeev, Denny Weinberg, ammar2
2016-09-15 19:16:43josh.rsetmessageid: <>
2016-09-15 19:16:43josh.rlinkissue27400 messages
2016-09-15 19:16:43josh.rcreate