We have hit an issue in the pickle module where the code throws an exception in a threaded environment:

The interesting piece of the backtrace is:

File "/xxx/1004060/lib/python3.7/site-packages/numpy/core/init.py", line 130, in _ufunc_reduce
return _ufunc_reconstruct, (whichmodule(func, name), name)
File "/xxx/lib/python3.7/pickle.py", line 309, in whichmodule
for module_name, module in list(sys.modules.items()):
RuntimeError: dictionary changed size during iteration

I tried to find a code path that would explain how the dict could be changed while the list is created but have not been able to find a code path that releases the GIL.

The executable is using many threads with imports happening in random threads and a custom class loader but we already make sure that the class loader is always holding the GIL.

The issue happens quite rarely (maybe once every one thousand's execution) so I don't have a reproducer right now.

Suggest making this change:

• for module_name, module in list(sys.modules.items()):
  + for module_name, module in sys.modules.copy().items():

We should consider making this change just about everywhere in the standard library. Besides avoiding a race condition, running a for loop over d.copy().items() is more memory efficient and possibly faster than a for loop over list(d.items()).

I afraid that this is a part of the larger issue (see also bpo-31165). Seems the cause is that the GIL can be released when allocate or reallocate the memory. Does your project use some memory managing hooks?

Using sys.modules.copy().items() can help a lot, because you allocate memory for new dict only once. In case of list(sys.modules.items()) you allocate memory for every key-value pair, and may allocate memory several times to resize a list.

This reduces the probability of occurring the problem in many times, but not to zero. There is still a chance that the size of the original dict increase when you allocate a memory for a new dict, so copying the data to a new dict can fail or even crash or spoil the memory.

New changeset 75bedbe by Raymond Hettinger in branch 'master':
bpo-40327: Improve atomicity, speed, and memory efficiency of the items() loop (GH-19628)
75bedbe

New changeset 16d0781 by Miss Islington (bot) in branch '3.8':
bpo-40327: Improve atomicity, speed, and memory efficiency of the items() loop (GH-19628)
16d0781

Does this really affect only python 3.7?

We see this in RHEL 8.2, using python 3.6.8:
https://bugzilla.redhat.com/show_bug.cgi?id=1837199#c69

Likely caused by:

    lvs = dict(self._lvs)

Without locking. self._lvs is a dict that may contain 1000's of items.

I'm not sure if this is relvant now for upstream, but backport to 3.6 would be useful.

Just to mention that we have seen similar issues whilst importing scipy in a multiprocessing context.
See scipy/scipy#11479 and https://stackoverflow.com/questions/68911221/python-runtimeerror-dictionary-changed-size-during-iteration-popping-out-rand.