This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: "bad magic number" when Python 2's pyc file exists without py file
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.11
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: mitchhentges, terry.reedy
Priority: normal Keywords: patch

Created on 2021-04-28 20:36 by mitchhentges, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
repro.sh mitchhentges, 2021-04-28 20:36 Script to reproduce the issue
Pull Requests
URL Status Linked Edit
PR 25731 closed mitchhentges, 2021-04-29 19:44
Messages (4)
msg392254 - (view) Author: Mitchell Hentges (mitchhentges) * Date: 2021-04-28 20:36
Python 3 imports may fail if a Python 2 .pyc file exists without an accompanying .py file.

-----

My project vendors code, and we recently updated "requests" to a drastically newer version.

As part of this version change, `requests/packages/__init__.py` was removed, and  `requests/packages.py` was added.

This project is long-lived, and many people have imported `requests` with Python 2. So, `requests/packages/__init__.pyc` exists.

After making this update, importing requests with Python 3 fails:
`ImportError: bad magic number in 'parent.child': b'\x03\xf3\r\n'`

Interestingly, deleting `requests/packages/__init__.pyc` allows Python 3 to work again.

-----

I've attached a "reproduction" script that produces the directory structure and tweak required to cause the failure.
I'm running Python 2.7.18 and Python 3.9.1.
msg392347 - (view) Author: Mitchell Hentges (mitchhentges) * Date: 2021-04-29 19:44
Looking closer into this, it appears that Python 3 will still try to load non-__pycache__ `.pyc` files. Perhaps I'm missing context, but aren't those only supposed by to written and read by Python 2?

Location where `pyc` loader is registered: https://github.com/python/cpython/blob/v3.9.4/Lib/importlib/_bootstrap_external.py#L1571-L1572
Location where `__pycache__` loading happens is part of SourceFileLoader here: https://github.com/python/cpython/blob/v3.9.4/Lib/importlib/_bootstrap_external.py#L870

I'll attach a patch to remove this pyc-loading for convenience, but I don't have enough context to be confident that it's the correct solution.
msg392521 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-04-30 22:33
Unless you find documentation that says otherwise, this an enhancement proposal for 3.11.

If no core developer (other than me) jumps on this, you might use git blame to fine out who committed code in the area you proposed changing.
msg405752 - (view) Author: Mitchell Hentges (mitchhentges) * Date: 2021-11-04 20:38
I'm going to close this here - I currently don't have the cycles to push this forward. Besides, this issue only occurs in environments that are using both Python 2 and 3, and as Python 2's EOL date gets further in the past, the severity of this ticket decreases.

For future readers, if you need a workaround for this exact problem, it was solved by my team with this import hook: https://hg.mozilla.org/mozilla-central/file/50e4a4e6975a7f828277a2b9c3cc9770a6c94107/build/mach_initialize.py#l630
History
Date User Action Args
2022-04-11 14:59:44adminsetgithub: 88135
2021-11-04 20:38:51mitchhentgessetstatus: open -> closed

messages: + msg405752
stage: patch review -> resolved
2021-04-30 22:33:00terry.reedysetversions: + Python 3.11, - Python 3.6, Python 3.7, Python 3.8, Python 3.9
nosy: + terry.reedy

messages: + msg392521

type: behavior -> enhancement
2021-04-29 19:44:40mitchhentgessetkeywords: + patch

stage: patch review
messages: + msg392347
pull_requests: + pull_request24422
2021-04-28 20:36:52mitchhentgescreate