classification
Title: test_venv and installation fail if SSL/TLS is not available
Type: behavior Stage: resolved
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: larry Nosy List: Dubslow, christian.heimes, dstufft, larry, ncoghlan, ned.deily, piotr.dobrogost, python-dev, tim.peters
Priority: release blocker Keywords: patch

Created on 2013-11-24 01:38 by tim.peters, last changed 2014-03-17 06:30 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue19744_ensurepip_no_ssl.diff ncoghlan, 2014-02-14 10:58 Minor tweak to prevent traceback in ensurepip CLI review
issue19744_ensurepip_install_ok_without_ssl.diff ncoghlan, 2014-02-14 11:11 Alternate approach that *ignores* missing SSL in the CLI review
Messages (28)
msg204149 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2013-11-24 01:38
With the current default branch, test_venv fails every time for me:

[1/1] test_venv
Traceback (most recent call last):
  File "C:\Code\Python\lib\runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Code\Python\lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "C:\Code\Python\lib\ensurepip\__main__.py", line 66, in <module>
    main()
  File "C:\Code\Python\lib\ensurepip\__main__.py", line 61, in main
    default_pip=args.default_pip,
  File "C:\Code\Python\lib\ensurepip\__init__.py", line 92, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "C:\Code\Python\lib\ensurepip\__init__.py", line 28, in _run_pip
    import pip
  File "C:\Users\Tim\AppData\Local\Temp\tmpjltqdgi8\pip-1.5.rc1-py2.py3-none-any.whl\pip\__init__.py", line 10
, in <module>
  File "C:\Users\Tim\AppData\Local\Temp\tmpjltqdgi8\pip-1.5.rc1-py2.py3-none-any.whl\pip\util.py", line 17, in
 <module>
  File "C:\Users\Tim\AppData\Local\Temp\tmpjltqdgi8\pip-1.5.rc1-py2.py3-none-any.whl\pip\_vendor\distlib\versi
on.py", line 14, in <module>
  File "C:\Users\Tim\AppData\Local\Temp\tmpjltqdgi8\pip-1.5.rc1-py2.py3-none-any.whl\pip\_vendor\distlib\compat.py", line 66, in <module>
ImportError: cannot import name 'HTTPSHandler'
test test_venv failed -- Traceback (most recent call last):
  File "C:\Code\Python\lib\test\test_venv.py", line 288, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
  File "C:\Code\Python\lib\test\test_venv.py", line 48, in run_with_capture
    func(*args, **kwargs)
  File "C:\Code\Python\lib\venv\__init__.py", line 359, in create
    builder.create(env_dir)
  File "C:\Code\Python\lib\venv\__init__.py", line 86, in create
    self._setup_pip(context)
  File "C:\Code\Python\lib\venv\__init__.py", line 242, in _setup_pip
    subprocess.check_output(cmd)
  File "C:\Code\Python\lib\subprocess.py", line 618, in check_output
    raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['C:\\Users\\Tim\\AppData\\Local\\Temp\\tmpt0ca1aqn\\Scripts\\python_d
.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

1 test failed:
    test_venv

All virtual Greek to me.
msg204152 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 01:51
Interesting - this isn't *quite* a duplicate of the buildbot failures in issue 19734 (at least, I don't think it is - the extra diagnostics I just checked in should tell us for sure).

Since pip isn't useful without HTTPS, we may want to add a check for ssl support to either venv or ensurepip itself.
msg204158 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2013-11-24 01:59
Ah, I didn't even notice the "S" in "HTTPS"!  I'm not building the SSL cruft on my box, so it's not surprising that anything requiring it would fail.  It is surprising that this is the only test that _does_ fail without it ;-)
msg204164 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2013-11-24 02:27
FYI, here's the new output:

[1/1] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "C:\Code\Python\lib\test\test_venv.py", line 289, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
subprocess.CalledProcessError: Command '['C:\\Users\\Tim\\AppData\\Local\\Temp\\tmptw2_vda6\\Scripts\\python_d
.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Code\Python\lib\test\test_venv.py", line 295, in test_with_pip
    self.fail(msg)
AssertionError: Command '['C:\\Users\\Tim\\AppData\\Local\\Temp\\tmptw2_vda6\\Scripts\\python_d.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

**Subprocess Output**
Traceback (most recent call last):
  File "C:\Code\Python\lib\runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Code\Python\lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "C:\Code\Python\lib\ensurepip\__main__.py", line 66, in <module>
    main()
  File "C:\Code\Python\lib\ensurepip\__main__.py", line 61, in main
    default_pip=args.default_pip,
  File "C:\Code\Python\lib\ensurepip\__init__.py", line 92, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "C:\Code\Python\lib\ensurepip\__init__.py", line 28, in _run_pip
    import pip
  File "C:\Users\Tim\AppData\Local\Temp\tmpdkfwi7it\pip-1.5.rc1-py2.py3-none-any.whl\pip\__init__.py", line 10
, in <module>
  File "C:\Users\Tim\AppData\Local\Temp\tmpdkfwi7it\pip-1.5.rc1-py2.py3-none-any.whl\pip\util.py", line 17, in
 <module>
  File "C:\Users\Tim\AppData\Local\Temp\tmpdkfwi7it\pip-1.5.rc1-py2.py3-none-any.whl\pip\_vendor\distlib\version.py", line 14, in <module>
  File "C:\Users\Tim\AppData\Local\Temp\tmpdkfwi7it\pip-1.5.rc1-py2.py3-none-any.whl\pip\_vendor\distlib\compat.py", line 66, in <module>
ImportError: cannot import name 'HTTPSHandler'


1 test failed:
    test_venv
msg204167 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-24 02:45
New changeset 9891ba920f3c by Nick Coghlan in branch 'default':
Issue #19744 (temp workaround): without ssl, skip pip test
http://hg.python.org/cpython/rev/9891ba920f3c
msg204168 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 02:49
Temporarily skipped the test to appease the build bots for the beta release, but this should be changed so that ensurepip refuses to bootstrap pip if SSL/TLS support is not available.

test_venv would then be updated to check for the appropriate behaviour (e.g. by inserting a dummy ssl.py that just raises import error into the venv)
msg204184 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 06:54
Also noting that the reason for the dummy ssl in the venv would be to provoke the "SSL/TLS not available" behaviour when running the tests in a Python that actually has those pieces (since the buildbots will have them available unless something goes wrong with the build)
msg204295 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-25 05:27
There's a ticket in pip to make pip work even when ssl isn't available. You wouldn't be able to install from PyPI but you would be able to install from local archives.
msg204312 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-25 08:38
Is that likely to be in 1.5 or 1.5.1? Not needing to special case this in
ensurepip would be nice :)
msg204328 - (view) Author: Bill Winslow (Dubslow) Date: 2013-11-25 11:50
I've stumbled upon what appears to be a related issue, but I'm not sure it deserves its own bug report.

I compiled 3.4 on my LMDE (so essentially Debian testing) system, and aside from not building tkinter, various compression modules, etc., all went well. I ran `make test` with no errors, including the success of test_venv. (There were many and various warnings about deprecated code and calling str() on bytes instances, but I'm pretty sure none of it is related).

I ran `sudo make altinstall` (to not nuke my current 3.3 from the repos), and to my surprise, it failed with the following error:




running install_scripts
copying build/scripts-3.4/pydoc3.4 -> /usr/local/bin
copying build/scripts-3.4/2to3-3.4 -> /usr/local/bin
copying build/scripts-3.4/idle3.4 -> /usr/local/bin
copying build/scripts-3.4/pyvenv-3.4 -> /usr/local/bin
changing mode of /usr/local/bin/pydoc3.4 to 755
changing mode of /usr/local/bin/2to3-3.4 to 755
changing mode of /usr/local/bin/idle3.4 to 755
changing mode of /usr/local/bin/pyvenv-3.4 to 755
rm /usr/local/lib/python3.4/lib-dynload/_sysconfigdata.py
rm -r /usr/local/lib/python3.4/lib-dynload/__pycache__
/usr/bin/install -c -m 644 ./Misc/python.man \
                /usr/local/share/man/man1/python3.4.1
if test "xupgrade" != "xno"  ; then \
                case upgrade in \
                  upgrade) ensurepip="--altinstall --upgrade" ;; \
                  install|*) ensurepip="--altinstall" ;; \
                esac; \
                 ./python -E -m ensurepip \
                  $ensurepip --root=/ ; \
        fi
Traceback (most recent call last):
  File "/home/bill/py3.4/Python-3.4.0b1/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/bill/py3.4/Python-3.4.0b1/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/home/bill/py3.4/Python-3.4.0b1/Lib/ensurepip/__main__.py", line 66, in <module>
    main()
  File "/home/bill/py3.4/Python-3.4.0b1/Lib/ensurepip/__main__.py", line 61, in main
    default_pip=args.default_pip,
  File "/home/bill/py3.4/Python-3.4.0b1/Lib/ensurepip/__init__.py", line 92, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/home/bill/py3.4/Python-3.4.0b1/Lib/ensurepip/__init__.py", line 28, in _run_pip
    import pip
  File "/tmp/tmprwpsemxj/pip-1.5.rc1-py2.py3-none-any.whl/pip/__init__.py", line 10, in <module>
  File "/tmp/tmprwpsemxj/pip-1.5.rc1-py2.py3-none-any.whl/pip/util.py", line 17, in <module>
  File "/tmp/tmprwpsemxj/pip-1.5.rc1-py2.py3-none-any.whl/pip/_vendor/distlib/version.py", line 14, in <module>
  File "/tmp/tmprwpsemxj/pip-1.5.rc1-py2.py3-none-any.whl/pip/_vendor/distlib/compat.py", line 66, in <module>
ImportError: cannot import name 'HTTPSHandler'
make: *** [altinstall] Error 1




Note: I will certainly *not* be trying to `sudo make install`.

In any case, the executable and modules were installed fine, so other than reporting it here, it's not causing me problems (so far).
msg204330 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-25 12:08
It probably can. I just need to figure out how to test it to make sure the PR that supposedly fixes it fixes it, and then figure out how to ensure it still works into the future.
msg204338 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-25 12:58
That I can help with. Steal the "import_fresh_module helper function from test.support (or the gist of it anyway - you can likely leave out the stuff about deprecated imports):

http://hg.python.org/cpython/file/default/Lib/test/support/__init__.py#l192


Then do:

   pip_nossl = import_fresh_module("pip", blocked=["ssl"])
msg204341 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-25 13:02
If the ssl import is actually in a submodule, you may need to list additional subpackages in the "fresh" parameter.

If you're wondering why this isn't in the importlib API - it's because it can go wrong in an impressively large number of ways, and we don't have any hope of documenting it well enough to make it usable by anyone that either: a) couldn't write it themselves; or b) isn't getting coached by someone that could write it themselves.

But when you know what you're doing and the modules involved don't break horribly when treated this way, it's a very neat trick for testing purposes.
msg205108 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-03 12:20
Relevant pip issue: https://github.com/pypa/pip/issues/1165
msg205817 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-12-10 14:41
Can this be solved in ensurepip for now? I've been banging away at this but it's going to require some refactoring in pip to make it reasonably work. The move to distlib and requests made this harder to do than the old PR against pip could handle.
msg206754 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-21 15:31
OK, since pip 1.5 will still have the SSL/TLS dependency, the approach I'll go with for 3.4 is to:

1. Have ensurepip refuse to bootstrap pip if the ssl module is not available (noting that we'll remove that restriction if pip 1.6 avoids the strict dependency)
2. Use import_fresh_module to check that behaviour
3. Ensure venv skips trying to bootstrap pip if the ssl module is not available (although the subprocess invocation in the venv tests could make that tricky to test when the ssl module actually *is* available)
msg206856 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-23 13:07
New changeset f670d8db8ef3 by Nick Coghlan in branch 'default':
Issue #19744: improve ensurepip error when ssl is missing
http://hg.python.org/cpython/rev/f670d8db8ef3
msg206857 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-23 13:14
I ended up not implementing step 3 - if you don't have SSL/TLS built, and you pass with_pip to the venv module API, or use the default settings for pyvenv, you *will* get an error from ensurepip.

Instead, I just kept the test skip in test_venv. ensurepip and test_ensurepip have been updated to provide a better traceback when SSL/TLS support is missing, though.

Tim, could you poke around at the latest version in your local build and see if the new checks are triggering? (I've assumed the ssl module can't be imported if the necessary underlying bits aren't built)
msg207164 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-01-02 13:34
This should be fixed, so I don't think it's a release blocker any more, but I also don't want to close it until Tim confirms it also works for him.
msg211024 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-11 21:19
This upsets "make install" as well - currently with a traceback.
msg211211 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-14 10:58
Attached patch tweaks the ensurepip CLI to check immediately for ssl support, and avoid printing a traceback if it is missing.

However, missing SSL support still fails the call, which fails installation.
msg211212 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-14 11:11
This patch is probably a better bet - it just prints a message to stderr to say that we're ignoring the ensurepip failure during installation.

pip wouldn't work anyway in a Python without ssl built, but at least this way that Python can still be installed without the ensurepip invocation complaining.

Ned, if this approach sounds reasonable to you, I'll commit this one.
msg211213 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-14 11:13
Note: I'm deliberately not worrying about ensurepip._uninstall here, since that only gets invoked implicitly in the Windows uninstaller, and that should always have a valid SSL to play with.

Always, since I worked out how to disable ssl in my local build (just a small tweak to setup.py), I was able to verify the correct behaviour of test_ensurepip and test_venv with SSL unavailable.
msg211222 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-02-14 17:03
issue19744_ensurepip_install_ok_without_ssl.diff looks good to me.  Note, though, that with it and with ssl support missing, test_ensurepip fails somewhat obscurely:

======================================================================
FAIL: test_bootstrap_version (test.test_ensurepip.TestBootstrappingMainFunction)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.4/test/test_ensurepip.py", line 293, in test_bootstrap_version
    ensurepip._main(["--version"])
AssertionError: SystemExit not raised

----------------------------------------------------------------------
msg211249 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-14 23:17
New changeset 9f76adbac8b7 by Nick Coghlan in branch 'default':
Issue #19744: Handle missing SSL/TLS in ensurepip
http://hg.python.org/cpython/rev/9f76adbac8b7
msg211250 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-14 23:22
Thanks Ned - I fixed that test to only run if SSL/TLS is available, and added a new one to test that the command "succeeds" (with a warning printed to stderr) if SSL/TLS is missing.

Larry - over to you to decide whether or not to cherry pick this into the release clone. The remaining misbehaviour that was fixed in the last patch only affects custom source builds, so the beneficiaries would be people trying to build from the source tarball or release tag without SSL/TLS support.
msg211620 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-19 13:42
Issue 20685 created to cover inclusion in 3.4.0, already committed to default, so closing this one.
msg213816 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-17 06:30
New changeset cd39d4cab680 by Nick Coghlan in branch '3.4':
Issue #19744: Handle missing SSL/TLS in ensurepip
http://hg.python.org/cpython/rev/cd39d4cab680
History
Date User Action Args
2014-03-17 06:30:58python-devsetmessages: + msg213816
2014-02-19 13:42:39ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg211620

stage: commit review -> resolved
2014-02-14 23:22:23ncoghlansetassignee: ned.deily -> larry
messages: + msg211250
2014-02-14 23:17:53python-devsetmessages: + msg211249
2014-02-14 17:03:19ned.deilysetmessages: + msg211222
2014-02-14 11:13:15ncoghlansetmessages: + msg211213
2014-02-14 11:11:19ncoghlansetfiles: + issue19744_ensurepip_install_ok_without_ssl.diff

nosy: + ned.deily
messages: + msg211212

assignee: ncoghlan -> ned.deily
2014-02-14 10:58:49ncoghlansetfiles: + issue19744_ensurepip_no_ssl.diff
messages: + msg211211

keywords: + patch
resolution: fixed -> (no value)
stage: resolved -> commit review
2014-02-11 21:25:09ncoghlansettitle: test_venv fails if SSL/TLS is not available -> test_venv and installation fail if SSL/TLS is not available
2014-02-11 21:24:31ncoghlansetassignee: tim.peters -> ncoghlan
2014-02-11 21:19:43ncoghlansetpriority: deferred blocker -> release blocker
status: pending -> open
messages: + msg211024
2014-02-04 12:34:08ncoghlansetstatus: open -> pending
2014-01-02 13:34:00ncoghlansetpriority: release blocker -> deferred blocker
status: pending -> open
messages: + msg207164
2013-12-23 13:14:02ncoghlansetstatus: open -> pending
type: behavior
messages: + msg206857

assignee: ncoghlan -> tim.peters
resolution: fixed
stage: resolved
2013-12-23 13:07:20python-devsetmessages: + msg206856
2013-12-21 15:31:01ncoghlansetassignee: ncoghlan
messages: + msg206754
2013-12-10 14:41:42dstufftsetmessages: + msg205817
2013-12-03 14:19:07piotr.dobrogostsetnosy: + piotr.dobrogost
2013-12-03 12:30:00ncoghlansettitle: ensurepip should refuse to install pip if SSL/TLS is not available -> test_venv fails if SSL/TLS is not available
2013-12-03 12:20:02ncoghlansetpriority: high -> release blocker
nosy: + larry
messages: + msg205108

2013-11-25 13:02:53ncoghlansetmessages: + msg204341
2013-11-25 12:58:05ncoghlansetmessages: + msg204338
2013-11-25 12:08:13dstufftsetmessages: + msg204330
2013-11-25 11:50:37Dubslowsetnosy: + Dubslow
messages: + msg204328
2013-11-25 08:38:14ncoghlansetmessages: + msg204312
2013-11-25 05:27:56dstufftsetnosy: + dstufft
messages: + msg204295
2013-11-24 06:54:23ncoghlansetmessages: + msg204184
2013-11-24 02:51:29christian.heimessetnosy: + christian.heimes
2013-11-24 02:49:32ncoghlansetpriority: normal -> high

messages: + msg204168
title: test_venv failing on 32-bit Windows Vista -> ensurepip should refuse to install pip if SSL/TLS is not available
2013-11-24 02:45:38python-devsetnosy: + python-dev
messages: + msg204167
2013-11-24 02:27:41tim.peterssetmessages: + msg204164
2013-11-24 01:59:32tim.peterssetmessages: + msg204158
2013-11-24 01:51:40ncoghlansetnosy: + ncoghlan
messages: + msg204152
2013-11-24 01:38:45tim.peterscreate