There appears to be a significant issue with importing from zip files on Python 3.3.4. I have tested this on Windows 7, 64-bit, using the MSI installer from python.org.

To reproduce, download virtualenv and extract the C:\Work\Scratch\virtualenv-pfm\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl file. Set PYTHONPATH to the full path of that wheel.

python -c "import pip" should work without error, but I see

PS 20:44 C:\Work\Scratch\virtualenv-pfm
>$env:PYTHONPATH="C:\Work\Scratch\virtualenv-pfm\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl"
PS 20:49 C:\Work\Scratch\virtualenv-pfm
>C:\Apps\Python33\python.exe -c "import pip"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'pip'

For comparison, the 32-bit version works perfectly:

PS 20:44 C:\Work\Scratch\virtualenv-pfm
>$env:PYTHONPATH="C:\Work\Scratch\virtualenv-pfm\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl"
PS 20:49 C:\Work\Scratch\virtualenv-pfm
>C:\Apps\Python33\python.exe -c "import pip"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'pip'

Apologies, cut & paste error.

3.3.2 32-bit:

PS 20:49 C:\Work\Scratch\virtualenv-pfm
>py -3.3-32 -c "import pip"
PS 20:49 C:\Work\Scratch\virtualenv-pfm
>py -3.3-32 -V
Python 3.3.2

3.4rc0 is not affected either:

PS 20:54 C:\Work\Scratch\virtualenv-pfm
>py -3.4 -c "import pip"
PS 20:57 C:\Work\Scratch\virtualenv-pfm
>py -3.4 -V
Python 3.4.0rc1

It fails for me on 3.4.0rc1 and 3.3.4:

PS D:\> $env:PYTHONPATH
D:\Python33_x64\Lib\site-packages\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl

PS D:\> py32 -V
Python 3.2.3
PS D:\> py32 -c "import pip; print(pip.__file__)"
D:\Python33_x64\Lib\site-packages\virtualenv_support\pip-1.5.2-py2.py3-none-any.whl\pip\__init__.py

PS D:\> py33 -V
Python 3.3.4
PS D:\> py33 -c "import pip; print(pip.__file__)"
D:\Python33\lib\site-packages\pip\__init__.py
(site-packages pip is after the wheel in sys.path)

PS D:\> py34 -V
Python 3.4.0rc1
PS D:\> py34 -c "import pip; print(pip.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'pip'

Could it be because zipimporter is not a complete loader?

>>> import zipimport, importlib.abc
>>> issubclass(zipimport.zipimporter, importlib.abc.Loader)
False
>>> pprint(set(dir(importlib.abc.Loader)) - set(dir(zipimport.zipimporter)))
{'__abstractmethods__',
 '__dict__',
 '__module__',
 '__weakref__',
 '_abc_cache',
 '_abc_negative_cache',
 '_abc_negative_cache_version',
 '_abc_registry',
 'create_module',
 'module_repr'}

gah.  i was tracking this down because I saw some strange errors that made me suspect it might exist (in 2.7) when trying to use a zipped up standard library but had not had time to confirm it. :(

I really should have rolled back the CLs causing it before 3.3.4 given that investigation was ongoing.  :(

We need to fix this breakage before 3.4.0 release.  The best thing to do for 3.4.0 is likely be to revert the following change:
 http://hg.python.org/cpython/rev/20b77ff040b6

For 3.3 the change that caused this is likely:
  http://hg.python.org/cpython/rev/2807a5f011e4

reverting that would restore the previous behavior.

Paul, are you in a position to try building 3.3.4 with that CL reverted to confirm?

We need better zipimport tests.
That test_zipimport passes while this problem exists is bad.

If you can come up with a reproducable test case for this, please go ahead and check it in to test_zipimport to highlight the brokenness.

I'm afraid I don't have things set up to do a Python build right now. I *might* have time later, and if so I'll give it a try, but I can't promise. Also, I haven't yet managed to show the issue on 3.4.0rc1, but that may be because my check wasn't as complete as Steve's.

OK, I have a build of 3.4 that shows the bug. How do I revert http://hg.python.org/cpython/rev/20b77ff040b6 ? hg backout -r 20b77ff040b6 says "abort: cannot backout a merge changeset".

I tried hg backout 2807a5f011e4. It gave errors in the form of dialog boxes titled "Microsoft Visual Studio" saying "File '-nosplash' not found". I've no idea where these came from and whether they matter, but I tried clicking OK. Mercurial then said

merging Lib/test/test_zipimport.py
 output file Lib/test/test_zipimport.py appears unchanged
was merge successful (yn)?

If I say "yes", the backout seems to complete, but building and testing again shows the problem still there.

So I'm stuck. That's about as much as I can do without some advice.

Note - we're going to have to work around this bug in Virtualenv (even if it's just in code specific to 3.3.4) so a workaround for 3.3.4 and an explanation of what's going wrong would be useful, as well as a fix.

I found that using zipimporter('...whl').load_module(pip) directly works fine, so you may be able to use that as a workaround. (Forgot to mention it in my initial comment, but that's what prompted me to check against the ABC.)

I also backed out that change from the latest 3.3, built and tried again but got the same result. That said, I'm not 100% sure my build is reliable, since I don't regularly build CPython right now. Here are my results:

Python 3.3.4+ (3.3:4330b3cb7245+, Feb 14 2014, 09:03:52) [MSC v.1800 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path[:] = r"D:\...\pip-1.5.2-py2.py3-none-any.whl"
>>> import pip
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'pip'


I also tried using zipimporter directly, which got further but still failed (probably because of my build - _ssl was not built):

>>> from zipimport import zipimporter
>>> zipimporter(r"D:\...\pip-1.5.2-py2.py3-none-any.whl").load_module('pip')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\__init__.py", line 10, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\util.py", line 18, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\_vendor\distlib\version.py", line 14, in <module>
  File "D:\...\pip-1.5.2-py2.py3-none-any.whl\pip\_vendor\distlib\compat.py", line 66, in <module>
ImportError: cannot import name HTTPSHandler

Oddly enough, this works with 3.3.4 as released:

>>> import sys, zipfile
>>> with zipfile.ZipFile('test.zip', 'w') as f:
...   f.writestr('test/__init__.py', b'print("Success!")')
...
>>> sys.path[:] = ['test.zip']
>>> import test
Success!

Maybe wheel is creating archives with settings that zipimport doesn't like? (Unless you invoke load_module directly...)

Thanks for the workaround. I'm going to try to do some serious testing of all this over the weekend, I'll let you know how it goes.

Paul, could you confirm that backing out the 2807a5f011e4 changeset on the 3.3 branch fixes the problem?

Georg - see http://bugs.python.org/issue20621#msg211209 for what I did. That shows backing out did *not* fix the issue. But I did backout on the default branch, and my backout may not have worked as I explained.

I just tried to do hg backout on the 3.3 branch and I got merge errors that I don't know how to fix.

I'm going to try hg checkout -r 88337 (that's the parent of the changeset you quoted) as am alternative approach. It'll take some time to set up the build tools I need. But if there's a better way of backing out as you want me to, let me know and I'll try that.

Yeah, just tried the backout, it doesn't apply cleanly anymore -- Modules/zipimport.c has apparently changed some more afterwards.

I think using the parent changeset is the best thing to do at the moment, until Greg comes up with a patch.

OK, confirmed.

Backing out to just before revision 2807a5f011e4 the problem has disappeared. Re-applying just 2807a5f011e4 causes the issue to appear again.

if you look at the zipimport.c file log there was one additional tiny change to fix a potential memory leak made after the change I suggested backing out:
http://hg.python.org/cpython/log/fb5f4cfc3230/Modules/zipimport.c

anyways, yes, backing up to the change before the ones i suggested backing out is the right thing to do.  patches attached though I don't have a reproduction case to confirm the problem and subsequent lack of problem yet.

OK, I can confirm that both the patches fix the issue when applied to the relevant branches.

OK. What do you think about rolling a 3.3.5 rc1 release next weekend, with the final a week later?

New changeset 5330de5c9d2d by Benjamin Peterson in branch '3.3':
backout 2807a5f011e4 for causing #20621
http://hg.python.org/cpython/rev/5330de5c9d2d

New changeset 52ab9e1ff46a by Benjamin Peterson in branch 'default':
merge backout for #20621
http://hg.python.org/cpython/rev/52ab9e1ff46a

New changeset 3dd8b0d31543 by Benjamin Peterson in branch '2.7':
backout #19081 to fix #20621
http://hg.python.org/cpython/rev/3dd8b0d31543

Okay, I backed out the world.

Confirmed that Modules/zipimport.c in the 3.3 and default branches is identical to the version from late last year.  Both should be good for 3.3.5 and 3.4.0 releases.

Can anyone confirm if this is a Windows-only issue? I need to know whether I should block Python 3.3.4 in virtualenv for everyone, or just for Windows.

I only have Windows I can test on.

I believe this happens on all platforms.

Will someone make sure this gets logged the way it needs to for Larry's 3.4rc2 cherrypick process? Ta.

Does anyone know how to write a unit test to ensure that the regression will not be reintroduced later?

I have reports from virtualenv users that this is not happening on Linux, BTW. Could it be platform-specific somehow? Have the Unix users here been able to reproduce this?

As regards a unit test, presumably we could ship a preconstructed zipfile in the test suite and just try importing from it. Although I'm surprised there isn't a test for this already...

Re unit tests. Doh. Of *course* there are tests for importing from zipfiles. And I just ran them (on Windows) and they worked. So there's something more complex going on here and I don't know how you test for it, sorry (short of by trying an import from the pip wheel that's bundled as part of _ensurepip - that's ugly and not very focused, but on the other hand it does test the precise use case that I have here...)

Thanks for trying. :)  I've got a complicated "test" case of zipping up the
stdlib into python27.zip and running the Python test suite against that
which also tends to trigger the bugs.  Some tests failing with SystemError
and such.

It smells like a memory corruption issue but I haven't had time to sit down
and dive in to find it. I won't proceed on my patches for the original bug
until I can figure it out and come up with a more exhaustive unittest that
demonstrates the bug.

Any progress to report?

there's a separate issue open for you with the necessary rollback patch to apply to your 3.4 release branch.  http://bugs.python.org/issue20651

This particular issue is actually solved in default, 3.3 and 2.7 as benjamin did the backouts/rollbacks of the problematic changes.

i'll be figuring out the bugs and redoing those only after finding appropriate test cases back in #19081.

Should this be mentioned in the 3.3.5 changelog (http://docs.python.org/3.3/whatsnew/changelog.html)?

New changeset 8899e4028561 by Gregory P. Smith in branch '3.3':
Mention issue 20621 fix in the NEWS file for 3.3.5rc1.
http://hg.python.org/cpython/rev/8899e4028561

New changeset 27b738209a60 by Benjamin Peterson in branch '3.4':
merge backout for #20621
http://hg.python.org/cpython/rev/27b738209a60