classification
Title: zipimport raises NameError: name '_boostrap_external' is not defined
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.9, Python 3.8
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: malin, misho88, miss-islington, ncoghlan, petr.viktorin, serhiy.storchaka, twouters, xtreak, zach.ware
Priority: normal Keywords: patch

Created on 2019-12-12 22:20 by misho88, last changed 2020-01-14 11:41 by petr.viktorin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17588 merged xtreak, 2019-12-13 13:30
PR 17642 merged xtreak, 2019-12-17 16:34
Messages (11)
msg358310 - (view) Author: Mihail Georgiev (misho88) Date: 2019-12-12 22:20
I think there's a "t" missing:

Lib/zipimport.py
609-
610-                try:
611:                    _boostrap_external._validate_hash_pyc(
612-                        data, source_hash, fullname, exc_details)
613-                except ImportError:
614-                    return None
615-    else:
616-        source_mtime, source_size = \
617-            _get_mtime_and_size_of_source(self, fullpath)
618-
619-        if source_mtime:
msg358311 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2019-12-12 22:48
Good catch!  Would you like to submit a pull request to fix it?  Ideally such a PR should also include a test to exercise this code.
msg358316 - (view) Author: Ma Lin (malin) * Date: 2019-12-13 05:28
Is it possible to scan stdlib to find similar bugs?
msg358317 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-12-13 06:08
A possible test case to trigger this code path based on testUncheckedHashBasedPyc would be as below. check_hash_based_pycs can be patched to be "always" so that the hash is validated using _bootstrap_external._validate_hash_pyc. The difference between "state = old" for pyc and "state = new" in py would ensure the hashes are different to raise ImportError expected and the test picks up state = new as the updated source code. Without fixing the typo trying to make hash validation would throw NameError.


@unittest.mock.patch('_imp.check_hash_based_pycs', 'always')
def test_checked_hash_based_change_pyc(self):
    source = b"state = 'old'"
    source_hash = importlib.util.source_hash(source)
    bytecode = importlib._bootstrap_external._code_to_hash_pyc(
        compile(source, "???", "exec"),
        source_hash,
        False,
    )
    files = {TESTMOD + ".py": (NOW, "state = 'new'"),
             TESTMOD + ".pyc": (NOW - 20, bytecode)}
    def check(mod):
        self.assertEqual(mod.state, 'new')
    self.doTest(None, files, TESTMOD, call=check)
msg358318 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-12-13 06:15
Also since the fix involves modifying zipimport.py it would also require running "make regen-importlib && make -s" for changes to ensure the updated regen code is used.
msg358319 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-12-13 06:19
Ma Lin, Running pylint/flake8 could possibly detect it but needs to be manually checked. issue36948 was a similar report I filed in the past using flake8 regarding NameError.

pylint - Lib/zipimport.py:611:20: E0602: Undefined variable '_boostrap_external' (undefined-variable)
flake8 - Lib/zipimport.py:611:21: F821 undefined name '_boostrap_external'
msg358321 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-12-13 06:51
Wince you has virtually written a patch, do you mind to create a PR Karthikeyan?
msg358326 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-12-13 10:47
Sure Serhiy, I will create one.
msg358459 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-12-15 23:34
New changeset 79f02fee1a542c440fd906fd54154c73fc0f8235 by Nick Coghlan (Xtreak) in branch 'master':
bpo-39033: Fix NameError in zipimport during hash validation (GH-17588)
https://github.com/python/cpython/commit/79f02fee1a542c440fd906fd54154c73fc0f8235
msg359964 - (view) Author: miss-islington (miss-islington) Date: 2020-01-14 11:39
New changeset 9955f33cdbf27de270038dfbad37d15b160ecca2 by Miss Islington (bot) (Karthikeyan Singaravelan) in branch '3.8':
[3.8] bpo-39033: Fix NameError in zipimport during hash validation (GH-17588) (GH-17642)
https://github.com/python/cpython/commit/9955f33cdbf27de270038dfbad37d15b160ecca2
msg359965 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2020-01-14 11:41
Thank you, Mihail and Karthikeyan!
History
Date User Action Args
2020-01-14 11:41:34petr.viktorinsetstatus: open -> closed

nosy: + petr.viktorin
messages: + msg359965

stage: patch review -> resolved
2020-01-14 11:39:23miss-islingtonsetnosy: + miss-islington
messages: + msg359964
2019-12-17 16:34:19xtreaksetpull_requests: + pull_request17111
2019-12-15 23:34:18ncoghlansetnosy: + ncoghlan
messages: + msg358459
2019-12-13 13:30:04xtreaksetkeywords: + patch
stage: patch review
pull_requests: + pull_request17060
2019-12-13 10:47:15xtreaksetmessages: + msg358326
2019-12-13 06:51:39serhiy.storchakasetmessages: + msg358321
2019-12-13 06:19:36xtreaksetmessages: + msg358319
2019-12-13 06:15:18xtreaksetmessages: + msg358318
2019-12-13 06:08:47xtreaksetnosy: + xtreak
messages: + msg358317
2019-12-13 05:28:34malinsetnosy: + malin
messages: + msg358316
2019-12-13 01:47:59xtreaksetnosy: + serhiy.storchaka
2019-12-12 22:48:44zach.waresetnosy: + zach.ware, twouters
messages: + msg358311
2019-12-12 22:20:17misho88create