classification
Title: Unexpected exception with zip importer
Type: behavior Stage: resolved
Components: Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, miss-islington, pablogsal, ronaldoussoren, twouters
Priority: normal Keywords: 3.10regression, patch

Created on 2021-09-13 11:29 by ronaldoussoren, last changed 2021-09-18 00:47 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
repro.py ronaldoussoren, 2021-09-13 11:29
Pull Requests
URL Status Linked Edit
PR 28435 merged brett.cannon, 2021-09-17 23:00
PR 28437 closed miss-islington, 2021-09-17 23:48
PR 28438 merged brett.cannon, 2021-09-18 00:14
Messages (8)
msg401698 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-09-13 11:29
The attached file demonstrates the problem:

If importlib.invalidate_caches() is called while the zipfile used by the zip importer is not available the import system breaks entirely. I found this in a testsuite that accedently did this (it should have updated sys.path). 

I get the following exception:

$ python3.10 t.py
Traceback (most recent call last):
  File "/Users/ronald/Projects/modulegraph2/t.py", line 27, in <module>
    import uu
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1430, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1402, in _get_spec
  File "<frozen zipimport>", line 168, in find_spec
  File "<frozen zipimport>", line 375, in _get_module_info
TypeError: argument of type 'NoneType' is not iterable


This exception is not very friendly....

This particular exception is caused by setting self._files to None in the importer's invalidate_caches method, while not checking for None in _get_modules_info. 

I'm not sure what the best fix would be, setting self._files to an empty list would likely be the easiest fix.

Note that the script runs without errors in Python 3.9.
msg401793 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-14 19:17
I bisected this to:

3abf6f010243a91bf57cbf357dac33193f7b8407 is the first bad commit
commit 3abf6f010243a91bf57cbf357dac33193f7b8407
Author: Desmond Cheong <desmondcheongzx@gmail.com>
Date:   Tue Mar 9 04:06:02 2021 +0800

    bpo-14678: Update zipimport to support importlib.invalidate_caches() (GH-24159)



    Added an invalidate_caches() method to the zipimport.zipimporter class based on the implementation of importlib.FileFinder.invalidate_caches(). This was done by adding a get_files() method and an _archive_mtime attribute to zipimport.zipimporter to check for updates or cache invalidation whenever the cache of files and toc entry information in the zipimporter is accessed.

 Doc/library/zipimport.rst                          |    9 +
 Lib/test/test_zipimport.py                         |   41 +
 Lib/zipimport.py                                   |   10 +
 .../2021-01-07-21-25-49.bpo-14678.1zniCH.rst       |    3 +
 Python/importlib_zipimport.h                       | 1896 ++++++++++----------
 5 files changed, 1020 insertions(+), 939 deletions(-)
 create mode 100644 bpo-14678.1zniCH.rst">Misc/NEWS.d/next/Library/2021-01-07-21-25-49.bpo-14678.1zniCH.rst
bisect run success


Which is https://bugs.python.org/issue14678
msg401794 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-14 19:17
Brett, can you take a look when you have some time?
msg401935 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-09-16 10:44
I just noticed that I'm unnecessarily obtuse in my description of a possible fix, the diff (without test update):

% git diff Lib/zipimport.py                                                                                        (main)cpython
diff --git a/Lib/zipimport.py b/Lib/zipimport.py
index c55fec6aa1..43ac6cbe57 100644
--- a/Lib/zipimport.py
+++ b/Lib/zipimport.py
@@ -334,7 +334,7 @@ def invalidate_caches(self):
             _zip_directory_cache[self.archive] = self._files
         except ZipImportError:
             _zip_directory_cache.pop(self.archive, None)
-            self._files = None
+            self._files = {}
 
 
     def __repr__(self):


With that change the exception should not happen, and the now stale zipimporter would be ignored when flushing the cache while the zipfile referenced by the zip importer instance has been removed.

That said, I haven't tested this and won't create a PR because my local tree is (still) a mess.
msg402109 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-09-17 23:07
The proposed fix seems to be the right one based on my reading of the code.
msg402111 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-09-17 23:48
New changeset 209b7035f714dcc41df054b0b023e0b955d7e1a2 by Brett Cannon in branch 'main':
bpo-45183: don't raise an exception when calling zipimport.zipimporter.find_spec() when the zip file is missing and the internal cache has been reset (GH-28435)
https://github.com/python/cpython/commit/209b7035f714dcc41df054b0b023e0b955d7e1a2
msg402114 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-09-18 00:46
New changeset e1bdecb6dc7ac33256d5fa875d45634512d2a90e by Brett Cannon in branch '3.10':
[3.10] bpo-45183: don't raise an exception when calling zipimport.zipimporter.find_spec() when the zip file is missing and the internal cache has been reset (GH-28435) (#28438)
https://github.com/python/cpython/commit/e1bdecb6dc7ac33256d5fa875d45634512d2a90e
msg402115 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-09-18 00:47
I decided that find_spec() saying something wasn't available in a finder made sense even though it was because a zip file no longer existed as the loader would still fail as appropriate.
History
Date User Action Args
2021-09-18 00:47:38brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg402115

stage: patch review -> resolved
2021-09-18 00:46:30brett.cannonsetmessages: + msg402114
2021-09-18 00:14:25brett.cannonsetpull_requests: + pull_request26843
2021-09-17 23:48:26brett.cannonsetmessages: + msg402111
2021-09-17 23:48:25miss-islingtonsetkeywords: + patch
nosy: + miss-islington

pull_requests: + pull_request26842
stage: patch review
2021-09-17 23:07:11brett.cannonsetkeywords: - patch

messages: + msg402109
stage: patch review -> (no value)
2021-09-17 23:00:27brett.cannonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request26840
2021-09-16 10:44:35ronaldoussorensetmessages: + msg401935
2021-09-15 19:23:14brett.cannonsetassignee: brett.cannon
2021-09-14 19:17:48pablogsalsetmessages: + msg401794
2021-09-14 19:17:30pablogsalsetnosy: + brett.cannon
messages: + msg401793
2021-09-13 11:35:41ronaldoussorensettype: behavior
2021-09-13 11:29:39ronaldoussorensetnosy: + twouters, pablogsal
2021-09-13 11:29:05ronaldoussorencreate