classification
Title: zipimport is not PEP 3147 or PEP 488 compliant
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.6
process
Status: open Resolution:
Dependencies: 25711 Superseder:
Assigned To: Nosy List: brett.cannon, byrnes, eric.snow, lukasz.langa, matrixise, ncoghlan, serhiy.storchaka, superluser, twouters
Priority: normal Keywords:

Created on 2015-11-23 15:44 by byrnes, last changed 2018-09-19 07:57 by serhiy.storchaka.

Messages (4)
msg255178 - (view) Author: Robert Byrnes (byrnes) Date: 2015-11-23 15:44
zipimport is not PEP 3147 compliant: i.e., it looks for foo.pyc (in the same directory as foo.py) instead of __pycache__/foo.cpython-35.pyc.  This is counterintuitive, and unfortunate because it means that installation directories (that obey PEP 3147 conventions) can't be zip archived from filesystems and then used by zipimport.

When support for PEP 488 was added to Python 3.5, zipimport was modified to just delete references to .pyo files, but it needs to know about the new .opt-[12] filename components (as well as the PEP 3147 version tags).  As far as I can tell, the current version of zipimport can't import optimized bytecode files, using either the old or new filename conventions.

Finally, none of this behavior is documented.  The zipimport docs still mention .pyo files (which were eliminated by PEP 488), and say nothing about the filename conventions expected within zip archives.
msg255184 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-11-23 16:29
Not finding bytecode files is not that big of a deal. While it's a very minor performance loss, it isn't critical to how Python works (and since it looks for bytecode-only files that use-case isn't broken either).

But the real problem is that zipimport is such a nasty chunk of code that no one wants to fix it. Yet more evidence we just need to rewrite the whole module. I created issue #25711 to track a rewrite.
msg259446 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2016-02-02 23:22
If you're affected, there's a workaround. Use the following script to rewrite paths in your .zip file to be zipimport-compatible:

https://gist.github.com/ambv/909d38bdac4f3e719b7c

It preserves the preamble and all file metadata.

A little context why this is useful: for archives that are effectively "static" environments with entire dependency trees in them, this bug causes both a big slowdown in startup time (counted in seconds), as well as tens of megabytes of .pyc created on the fly. Examples include zipapp-style .pyz, Twitter's .pex or Facebook's .par. 

So while we're waiting for a new zipimport implementation, the workaround listed above should unblock people uzing this form of Python package distribution.
msg325732 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-19 07:57
zipimport has been rewritten in pure Python (issue25711).
History
Date User Action Args
2018-09-19 07:57:03serhiy.storchakasetmessages: + msg325732
2016-02-02 23:22:51lukasz.langasetnosy: + lukasz.langa
messages: + msg259446
2015-11-23 20:33:48eric.snowsetnosy: + superluser
2015-11-23 18:04:19matrixisesetnosy: + matrixise
2015-11-23 16:29:06brett.cannonsetdependencies: + Rewrite zipimport from scratch
messages: + msg255184
2015-11-23 15:52:03serhiy.storchakasetnosy: + twouters, brett.cannon, ncoghlan, eric.snow, serhiy.storchaka
2015-11-23 15:44:56byrnescreate