classification
Title: checked hash-based pyc files not working with imp module
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, brett.cannon, miss-islington, phmccarty
Priority: normal Keywords: patch

Created on 2018-07-05 22:31 by phmccarty, last changed 2018-07-07 04:00 by miss-islington. This issue is now closed.

Files
File name Uploaded Description Edit
imp-test.py phmccarty, 2018-07-05 22:31 imp test code
imp-test-mod.py phmccarty, 2018-07-05 22:32 imp test module
Pull Requests
URL Status Linked Edit
PR 8130 merged benjamin.peterson, 2018-07-06 05:46
PR 8146 merged miss-islington, 2018-07-07 03:41
Messages (6)
msg321139 - (view) Author: Patrick McCarty (phmccarty) Date: 2018-07-05 22:31
OS: Clear Linux build 23460
Python version: 3.7.0

Description:
I am seeing an uncaught exception in Python 3.7.0 when using the "imp" module to import a module that has a checked hash-based pyc file. See the attached source files.

Steps to reproduce:
1) Copy attached source files to a directory
2) In that directory, run
    $ python3.7 -m compileall --invalidation-mode checked-hash imp-test-mod.py
    $ python3.7 imp-test.py
3) See the resulting output (omitting the imp deprecation notice):

Traceback (most recent call last):
  File "imp-test.py", line 6, in <module>
    mod = imp.load_module(modname, f, p, d)
  File "/usr/lib/python3.7/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.7/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 838, in get_code
TypeError: a bytes-like object is required, not 'str'
msg321147 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-07-06 04:52
Can you not use imp? It's been deprecated since 3.4 and gradually become more and more broken and useless as this and other import improvements (e.g., namespace packages) have happened.
msg321149 - (view) Author: Patrick McCarty (phmccarty) Date: 2018-07-06 05:18
Thanks for the response.

I would like to avoid using imp, but I work on a distro team that ships many packages that still use the module. We wanted to start using checked-hash invalidation right away after upgrading to 3.7.0, but some of our release tests failed due to this issue.

I am willing to file bugs against other projects still using imp to stop using it, if that is the recommended direction. Considering that imp still works with timestamp invalidation, and checked-hash invalidation being a brand new feature, I wanted to file a bug here first before reaching out to the other projects.
msg321150 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-07-06 05:42
I see. Thanks for being an early adopter! We can probably hack something together for imp. I'll send a PR in a moment.

Filing bugs about imp usage would be valuable, especially if it flushes out any cases where imp provides functionality that our newer APIs don't. We'd quite like to kill imp.
msg321202 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-07-07 03:41
New changeset b0274f2cddd36b49fe5080efbe160277ef546471 by Benjamin Peterson in branch 'master':
closes bpo-34056: Always return bytes from _HackedGetData.get_data(). (GH-8130)
https://github.com/python/cpython/commit/b0274f2cddd36b49fe5080efbe160277ef546471
msg321203 - (view) Author: miss-islington (miss-islington) Date: 2018-07-07 04:00
New changeset 7bd6f0e5500f778e940374237b94651f60ae1990 by Miss Islington (bot) in branch '3.7':
closes bpo-34056: Always return bytes from _HackedGetData.get_data(). (GH-8130)
https://github.com/python/cpython/commit/7bd6f0e5500f778e940374237b94651f60ae1990
History
Date User Action Args
2018-07-07 04:00:48miss-islingtonsetnosy: + miss-islington
messages: + msg321203
2018-07-07 03:41:33miss-islingtonsetpull_requests: + pull_request7716
2018-07-07 03:41:11benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg321202

stage: patch review -> resolved
2018-07-06 05:46:34benjamin.petersonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request7708
2018-07-06 05:42:24benjamin.petersonsetnosy: + brett.cannon
messages: + msg321150
2018-07-06 05:18:00phmccartysetmessages: + msg321149
2018-07-06 04:52:34benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg321147
2018-07-05 22:32:11phmccartysetfiles: + imp-test-mod.py
2018-07-05 22:31:12phmccartycreate