classification
Title: Issue with zipimport in 3.3.4 and 3.4.0rc1
Type: behavior Stage:
Components: Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: Arfrever, benjamin.peterson, georg.brandl, gregory.p.smith, haypo, larry, pmoore, python-dev, steve.dower
Priority: release blocker Keywords: 3.3regression, patch

Created on 2014-02-13 20:50 by pmoore, last changed 2014-03-17 06:31 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue20621-33-rollbacks-gps01.diff gregory.p.smith, 2014-02-16 18:25 review
issue20621-34-rollbacks-gps01.diff gregory.p.smith, 2014-02-16 18:26 review
Messages (33)
msg211170 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-13 20:50
There appears to be a significant issue with importing from zip files on Python 3.3.4. I have tested this on Windows 7, 64-bit, using the MSI installer from python.org.

To reproduce, download virtualenv and extract the C:\Work\Scratch\virtualenv-pfm\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl file. Set PYTHONPATH to the full path of that wheel.

python -c "import pip" should work without error, but I see

PS 20:44 C:\Work\Scratch\virtualenv-pfm
>$env:PYTHONPATH="C:\Work\Scratch\virtualenv-pfm\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl"
PS 20:49 C:\Work\Scratch\virtualenv-pfm
>C:\Apps\Python33\python.exe -c "import pip"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'pip'

For comparison, the 32-bit version works perfectly:

PS 20:44 C:\Work\Scratch\virtualenv-pfm
>$env:PYTHONPATH="C:\Work\Scratch\virtualenv-pfm\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl"
PS 20:49 C:\Work\Scratch\virtualenv-pfm
>C:\Apps\Python33\python.exe -c "import pip"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'pip'
msg211172 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-13 21:11
Apologies, cut & paste error.

3.3.2 32-bit:

PS 20:49 C:\Work\Scratch\virtualenv-pfm
>py -3.3-32 -c "import pip"
PS 20:49 C:\Work\Scratch\virtualenv-pfm
>py -3.3-32 -V
Python 3.3.2

3.4rc0 is not affected either:

PS 20:54 C:\Work\Scratch\virtualenv-pfm
>py -3.4 -c "import pip"
PS 20:57 C:\Work\Scratch\virtualenv-pfm
>py -3.4 -V
Python 3.4.0rc1
msg211190 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-02-13 23:20
It fails for me on 3.4.0rc1 and 3.3.4:

PS D:\> $env:PYTHONPATH
D:\Python33_x64\Lib\site-packages\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl

PS D:\> py32 -V
Python 3.2.3
PS D:\> py32 -c "import pip; print(pip.__file__)"
D:\Python33_x64\Lib\site-packages\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl\pip\__init__.py

PS D:\> py33 -V
Python 3.3.4
PS D:\> py33 -c "import pip; print(pip.__file__)"
D:\Python33\lib\site-packages\pip\__init__.py
(site-packages pip is after the wheel in sys.path)

PS D:\> py34 -V
Python 3.4.0rc1
PS D:\> py34 -c "import pip; print(pip.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'pip'

Could it be because zipimporter is not a complete loader?

>>> import zipimport, importlib.abc
>>> issubclass(zipimport.zipimporter, importlib.abc.Loader)
False
>>> pprint(set(dir(importlib.abc.Loader)) - set(dir(zipimport.zipimporter)))
{'__abstractmethods__',
 '__dict__',
 '__module__',
 '__weakref__',
 '_abc_cache',
 '_abc_negative_cache',
 '_abc_negative_cache_version',
 '_abc_registry',
 'create_module',
 'module_repr'}
msg211194 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-02-14 01:58
gah.  i was tracking this down because I saw some strange errors that made me suspect it might exist (in 2.7) when trying to use a zipped up standard library but had not had time to confirm it. :(

I really should have rolled back the CLs causing it before 3.3.4 given that investigation was ongoing.  :(

We need to fix this breakage before 3.4.0 release.  The best thing to do for 3.4.0 is likely be to revert the following change:
 http://hg.python.org/cpython/rev/20b77ff040b6

For 3.3 the change that caused this is likely:
  http://hg.python.org/cpython/rev/2807a5f011e4

reverting that would restore the previous behavior.

Paul, are you in a position to try building 3.3.4 with that CL reverted to confirm?

We need better zipimport tests.
That test_zipimport passes while this problem exists is bad.

If you can come up with a reproducable test case for this, please go ahead and check it in to test_zipimport to highlight the brokenness.
msg211206 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-14 08:05
I'm afraid I don't have things set up to do a Python build right now. I *might* have time later, and if so I'll give it a try, but I can't promise. Also, I haven't yet managed to show the issue on 3.4.0rc1, but that may be because my check wasn't as complete as Steve's.
msg211208 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-14 10:01
OK, I have a build of 3.4 that shows the bug. How do I revert http://hg.python.org/cpython/rev/20b77ff040b6 ? hg backout -r 20b77ff040b6 says "abort: cannot backout a merge changeset".
msg211209 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-14 10:28
I tried hg backout 2807a5f011e4. It gave errors in the form of dialog boxes titled "Microsoft Visual Studio" saying "File '-nosplash' not found". I've no idea where these came from and whether they matter, but I tried clicking OK. Mercurial then said

merging Lib/test/test_zipimport.py
 output file Lib/test/test_zipimport.py appears unchanged
was merge successful (yn)?

If I say "yes", the backout seems to complete, but building and testing again shows the problem still there.

So I'm stuck. That's about as much as I can do without some advice.
msg211219 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-14 13:32
Note - we're going to have to work around this bug in Virtualenv (even if it's just in code specific to 3.3.4) so a workaround for 3.3.4 and an explanation of what's going wrong would be useful, as well as a fix.
msg211223 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-02-14 17:12
I found that using zipimporter('...whl').load_module(pip) directly works fine, so you may be able to use that as a workaround. (Forgot to mention it in my initial comment, but that's what prompted me to check against the ABC.)

I also backed out that change from the latest 3.3, built and tried again but got the same result. That said, I'm not 100% sure my build is reliable, since I don't regularly build CPython right now. Here are my results:

Python 3.3.4+ (3.3:4330b3cb7245+, Feb 14 2014, 09:03:52) [MSC v.1800 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path[:] = r"D:\...\pip-1.5.2-py2.py3-none-any.whl"
>>> import pip
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'pip'


I also tried using zipimporter directly, which got further but still failed (probably because of my build - _ssl was not built):

>>> from zipimport import zipimporter
>>> zipimporter(r"D:\...\pip-1.5.2-py2.py3-none-any.whl").load_module('pip')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\__init__.py", line 10, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\util.py", line 18, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\_vendor\distlib\version.py", line 14, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\_vendor\distlib\compat.py", line 66, in <module>
ImportError: cannot import name HTTPSHandler
msg211224 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-02-14 17:19
Oddly enough, this works with 3.3.4 as released:

>>> import sys, zipfile
>>> with zipfile.ZipFile('test.zip', 'w') as f:
...   f.writestr('test/__init__.py', b'print("Success!")')
...
>>> sys.path[:] = ['test.zip']
>>> import test
Success!

Maybe wheel is creating archives with settings that zipimport doesn't like? (Unless you invoke load_module directly...)
msg211228 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-14 17:52
Thanks for the workaround. I'm going to try to do some serious testing of all this over the weekend, I'll let you know how it goes.
msg211307 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-02-16 08:21
Paul, could you confirm that backing out the 2807a5f011e4 changeset on the 3.3 branch fixes the problem?
msg211308 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-16 08:53
Georg - see http://bugs.python.org/issue20621#msg211209 for what I did. That shows backing out did *not* fix the issue. But I did backout on the default branch, and my backout may not have worked as I explained.

I just tried to do hg backout on the 3.3 branch and I got merge errors that I don't know how to fix.

I'm going to try hg checkout -r 88337 (that's the parent of the changeset you quoted) as am alternative approach. It'll take some time to set up the build tools I need. But if there's a better way of backing out as you want me to, let me know and I'll try that.
msg211309 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-02-16 09:03
Yeah, just tried the backout, it doesn't apply cleanly anymore -- Modules/zipimport.c has apparently changed some more afterwards.

I think using the parent changeset is the best thing to do at the moment, until Greg comes up with a patch.
msg211313 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-16 11:32
OK, confirmed.

Backing out to just before revision 2807a5f011e4 the problem has disappeared. Re-applying just 2807a5f011e4 causes the issue to appear again.
msg211335 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-02-16 18:25
if you look at the zipimport.c file log there was one additional tiny change to fix a potential memory leak made after the change I suggested backing out:
http://hg.python.org/cpython/log/fb5f4cfc3230/Modules/zipimport.c

anyways, yes, backing up to the change before the ones i suggested backing out is the right thing to do.  patches attached though I don't have a reproduction case to confirm the problem and subsequent lack of problem yet.
msg211341 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-16 18:56
OK, I can confirm that both the patches fix the issue when applied to the relevant branches.
msg211342 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-02-16 19:02
OK. What do you think about rolling a 3.3.5 rc1 release next weekend, with the final a week later?
msg211345 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-16 19:21
New changeset 5330de5c9d2d by Benjamin Peterson in branch '3.3':
backout 2807a5f011e4 for causing #20621
http://hg.python.org/cpython/rev/5330de5c9d2d

New changeset 52ab9e1ff46a by Benjamin Peterson in branch 'default':
merge backout for #20621
http://hg.python.org/cpython/rev/52ab9e1ff46a

New changeset 3dd8b0d31543 by Benjamin Peterson in branch '2.7':
backout #19081 to fix #20621
http://hg.python.org/cpython/rev/3dd8b0d31543
msg211346 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-02-16 19:22
Okay, I backed out the world.
msg211355 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-02-16 20:10
Confirmed that Modules/zipimport.c in the 3.3 and default branches is identical to the version from late last year.  Both should be good for 3.3.5 and 3.4.0 releases.
msg211366 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-16 22:43
Can anyone confirm if this is a Windows-only issue? I need to know whether I should block Python 3.3.4 in virtualenv for everyone, or just for Windows.

I only have Windows I can test on.
msg211367 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-02-16 22:44
I believe this happens on all platforms.
msg211377 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-16 23:46
Will someone make sure this gets logged the way it needs to for Larry's 3.4rc2 cherrypick process? Ta.
msg211380 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-02-16 23:57
Does anyone know how to write a unit test to ensure that the regression will not be reintroduced later?
msg211396 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-17 07:27
I have reports from virtualenv users that this is not happening on Linux, BTW. Could it be platform-specific somehow? Have the Unix users here been able to reproduce this?

As regards a unit test, presumably we could ship a preconstructed zipfile in the test suite and just try importing from it. Although I'm surprised there isn't a test for this already...
msg211397 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-17 07:36
Re unit tests. Doh. Of *course* there are tests for importing from zipfiles. And I just ran them (on Windows) and they worked. So there's something more complex going on here and I don't know how you test for it, sorry (short of by trying an import from the pip wheel that's bundled as part of _ensurepip - that's ugly and not very focused, but on the other hand it does test the precise use case that I have here...)
msg211437 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-02-17 19:13
Thanks for trying. :)  I've got a complicated "test" case of zipping up the
stdlib into python27.zip and running the Python test suite against that
which also tends to trigger the bugs.  Some tests failing with SystemError
and such.

It smells like a memory corruption issue but I haven't had time to sit down
and dive in to find it. I won't proceed on my patches for the original bug
until I can figure it out and come up with a more exhaustive unittest that
demonstrates the bug.
msg211692 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-02-20 06:05
Any progress to report?
msg211720 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-02-20 12:04
there's a separate issue open for you with the necessary rollback patch to apply to your 3.4 release branch.  http://bugs.python.org/issue20651

This particular issue is actually solved in default, 3.3 and 2.7 as benjamin did the backouts/rollbacks of the problematic changes.

i'll be figuring out the bugs and redoing those only after finding appropriate test cases back in #19081.
msg212173 - (view) Author: Paul Moore (pmoore) * Date: 2014-02-25 12:59
Should this be mentioned in the 3.3.5 changelog (http://docs.python.org/3.3/whatsnew/changelog.html)?
msg212232 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-26 00:46
New changeset 8899e4028561 by Gregory P. Smith in branch '3.3':
Mention issue 20621 fix in the NEWS file for 3.3.5rc1.
http://hg.python.org/cpython/rev/8899e4028561
msg213825 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-17 06:31
New changeset 27b738209a60 by Benjamin Peterson in branch '3.4':
merge backout for #20621
http://hg.python.org/cpython/rev/27b738209a60
History
Date User Action Args
2014-03-17 06:31:04python-devsetmessages: + msg213825
2014-03-04 22:54:39ned.deilylinkissue20852 superseder
2014-02-26 00:46:56python-devsetmessages: + msg212232
2014-02-25 12:59:21pmooresetmessages: + msg212173
2014-02-20 12:04:16gregory.p.smithsetstatus: open -> closed
resolution: fixed
messages: + msg211720
2014-02-20 06:05:49larrysetmessages: + msg211692
2014-02-17 19:13:01gregory.p.smithsetmessages: + msg211437
2014-02-17 07:36:09pmooresetmessages: + msg211397
2014-02-17 07:27:44pmooresetmessages: + msg211396
2014-02-16 23:57:09hayposetnosy: + haypo
messages: + msg211380
2014-02-16 23:46:05pmooresetmessages: + msg211377
2014-02-16 22:44:28gregory.p.smithsetmessages: + msg211367
2014-02-16 22:43:25pmooresetmessages: + msg211366
2014-02-16 20:10:19gregory.p.smithsetmessages: + msg211355
2014-02-16 19:22:28benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg211346
2014-02-16 19:21:45python-devsetnosy: + python-dev
messages: + msg211345
2014-02-16 19:02:13georg.brandlsetmessages: + msg211342
2014-02-16 18:56:23pmooresetmessages: + msg211341
2014-02-16 18:26:07gregory.p.smithsetfiles: + issue20621-34-rollbacks-gps01.diff
2014-02-16 18:25:52gregory.p.smithsetfiles: + issue20621-33-rollbacks-gps01.diff
keywords: + patch
messages: + msg211335
2014-02-16 11:32:07pmooresetmessages: + msg211313
2014-02-16 09:03:07georg.brandlsetassignee: gregory.p.smith
messages: + msg211309
2014-02-16 08:53:28pmooresetmessages: + msg211308
2014-02-16 08:21:45georg.brandlsetmessages: + msg211307
2014-02-14 17:52:16pmooresetmessages: + msg211228
2014-02-14 17:19:14steve.dowersetmessages: + msg211224
2014-02-14 17:12:31steve.dowersetmessages: + msg211223
2014-02-14 13:32:03pmooresetmessages: + msg211219
2014-02-14 10:28:26pmooresetmessages: + msg211209
2014-02-14 10:01:14pmooresetmessages: + msg211208
2014-02-14 08:05:24pmooresetmessages: + msg211206
2014-02-14 02:44:05Arfreversetnosy: + Arfrever
2014-02-14 01:58:58gregory.p.smithsettitle: Issue with zipimport in 3.3.4 -> Issue with zipimport in 3.3.4 and 3.4.0rc1
2014-02-14 01:58:15gregory.p.smithsetpriority: critical -> release blocker
versions: + Python 3.4
nosy: + larry

messages: + msg211194
2014-02-13 23:20:03steve.dowersetnosy: + steve.dower
messages: + msg211190
2014-02-13 23:18:05ncoghlansetnosy: + gregory.p.smith
2014-02-13 21:11:12pmooresetmessages: + msg211172
2014-02-13 20:50:36pmoorecreate