classification
Title: "bad magic number" when Python 2's pyc file exists without py file
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: mitchhentges, terry.reedy
Priority: normal Keywords: patch

Created on 2021-04-28 20:36 by mitchhentges, last changed 2021-04-30 22:33 by terry.reedy.

Files
File name Uploaded Description Edit
repro.sh mitchhentges, 2021-04-28 20:36 Script to reproduce the issue
Pull Requests
URL Status Linked Edit
PR 25731 closed mitchhentges, 2021-04-29 19:44
Messages (3)
msg392254 - (view) Author: Mitchell Hentges (mitchhentges) * Date: 2021-04-28 20:36
Python 3 imports may fail if a Python 2 .pyc file exists without an accompanying .py file.

-----

My project vendors code, and we recently updated "requests" to a drastically newer version.

As part of this version change, `requests/packages/__init__.py` was removed, and  `requests/packages.py` was added.

This project is long-lived, and many people have imported `requests` with Python 2. So, `requests/packages/__init__.pyc` exists.

After making this update, importing requests with Python 3 fails:
`ImportError: bad magic number in 'parent.child': b'\x03\xf3\r\n'`

Interestingly, deleting `requests/packages/__init__.pyc` allows Python 3 to work again.

-----

I've attached a "reproduction" script that produces the directory structure and tweak required to cause the failure.
I'm running Python 2.7.18 and Python 3.9.1.
msg392347 - (view) Author: Mitchell Hentges (mitchhentges) * Date: 2021-04-29 19:44
Looking closer into this, it appears that Python 3 will still try to load non-__pycache__ `.pyc` files. Perhaps I'm missing context, but aren't those only supposed by to written and read by Python 2?

Location where `pyc` loader is registered: https://github.com/python/cpython/blob/v3.9.4/Lib/importlib/_bootstrap_external.py#L1571-L1572
Location where `__pycache__` loading happens is part of SourceFileLoader here: https://github.com/python/cpython/blob/v3.9.4/Lib/importlib/_bootstrap_external.py#L870

I'll attach a patch to remove this pyc-loading for convenience, but I don't have enough context to be confident that it's the correct solution.
msg392521 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-04-30 22:33
Unless you find documentation that says otherwise, this an enhancement proposal for 3.11.

If no core developer (other than me) jumps on this, you might use git blame to fine out who committed code in the area you proposed changing.
History
Date User Action Args
2021-04-30 22:33:00terry.reedysetversions: + Python 3.11, - Python 3.6, Python 3.7, Python 3.8, Python 3.9
nosy: + terry.reedy

messages: + msg392521

type: behavior -> enhancement
2021-04-29 19:44:40mitchhentgessetkeywords: + patch

stage: patch review
messages: + msg392347
pull_requests: + pull_request24422
2021-04-28 20:36:52mitchhentgescreate