classification
Title: PEP 476: verify HTTPS certificates by default
Type: enhancement Stage:
Components: Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: 22366 Superseder:
Assigned To: alex Nosy List: Arfrever, alex, benjamin.peterson, christian.heimes, dstufft, jcea, jwilk, koobs, larry, ncoghlan, orsenthil, python-dev, raulcd
Priority: deferred blocker Keywords: needs review, patch

Created on 2014-09-15 12:34 by ncoghlan, last changed 2014-11-24 04:58 by benjamin.peterson. This issue is now closed.

Files
File name Uploaded Description Edit
pep476_minimal_implementation.diff ncoghlan, 2014-09-15 12:34 Minimal changes to verify HTTPS by default review
issue22417.diff alex, 2014-10-13 17:27 review
issue22417.diff alex, 2014-10-30 19:00 review
issue22417.diff alex, 2014-10-31 01:32 review
issue22417.diff alex, 2014-11-01 04:18 review
issue22417.diff alex, 2014-11-02 18:13 review
issue22417.diff alex, 2014-11-03 15:28 review
issue22417.diff alex, 2014-11-03 17:50 review
koobs-freebsd9.python3x-build2357.log koobs, 2014-11-13 23:20
issue22417.diff alex, 2014-11-23 23:30
Messages (16)
msg226912 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-09-15 12:34
Attached minimal patch updates http.client.HTTPSConnection to validate certs by default and adjusts test.test_httplib accordingly.

It doesn't currently include any docs changes, or changes to urllib.

The process wide "revert to the old behaviour" hook is to monkeypatch the ssl module:

    ssl._create_default_https_context = ssl._create_unverified_context

To monkeypatch the stdlib to validate *everything* (this one isn't new, just noting it for the record):

    ssl._create_stdlib_context = ssl.create_default_context
msg227093 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-09-19 13:38
Currently marking as a deferred blocker, as Alex wasn't sure he'd be able to get PEP 476 fully updated in time for 3.4.2rc1, and was willing to accept waiting for 2.7.9 and 3.4.3 rather than delaying 3.4.2 any further.

However, that was before Senthil accepted the patch in 22366 for 3.5, which means we're at "feature complete" for the proposed changes.

There's still the issue 22366 backport patch, PEP update, docs updates and What's New updates to go, so assigning to Alex to decide if he wants to work with Larry to get this ready to go for 3.4.2 (noting that the PEP still needs the final tick of approval from Guido after being updated to reflect the proposed implementation).

Otherwise we can get it ready for 2.7.9 with the other SSL changes, and it will appear in the 3.4.3 maintenance release, rather than 3.4.2.

(Note that I'm busy most of this weekend, so +1 from me in advance if you decide to go ahead with getting it into 3.4.2)
msg229255 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-10-13 17:27
Patch with the implementation, and initial work on documentation. Needs review please, I suspect we need more docs in more places. Feedback please!
msg230288 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-10-30 19:00
Patch now makes more precise assertions about the type of error that's occurring.
msg230317 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-10-31 01:32
Updates to teh docs based on teh feedback from Antoine.
msg230428 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-11-01 04:18
New version of the patch based on feedback from benjamin, should make it easier to do the 3.4 branch stuff.
msg230512 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-11-02 18:13
New patch uses self-signed.pythontest.net, instead of svn.python.org. svn.python.org is signed by CACert, which is in the root on some machines.
msg230541 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-11-03 15:21
% ./python Lib/test/regrtest.py -v test_urllib2_localnet
== CPython 3.4.2+ (3.4:7be6ef737aaf+, Nov 3 2014, 10:03:11) [GCC 4.8.3]
==   Linux-3.16.5-gentoo-x86_64-Intel-R-_Core-TM-_i7-2860QM_CPU_@_2.50GHz-with-gentoo-2.2 little-endian
==   hash algorithm: siphash24 64bit
==   /home/benjamin/dev/python/3.4/build/test_python_28724
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
[1/1] test_urllib2_localnet
test_basic_auth_httperror (test.test_urllib2_localnet.BasicAuthTests) ... ok
test_basic_auth_success (test.test_urllib2_localnet.BasicAuthTests) ... ok
test_proxy_qop_auth_int_works_or_throws_urlerror (test.test_urllib2_localnet.ProxyAuthTests) ... ok
test_proxy_qop_auth_works (test.test_urllib2_localnet.ProxyAuthTests) ... ok
test_proxy_with_bad_password_raises_httperror (test.test_urllib2_localnet.ProxyAuthTests) ... ok
test_proxy_with_no_password_raises_httperror (test.test_urllib2_localnet.ProxyAuthTests) ... ok
test_200 (test.test_urllib2_localnet.TestUrlopen) ... ok
test_200_with_parameters (test.test_urllib2_localnet.TestUrlopen) ... ok
test_404 (test.test_urllib2_localnet.TestUrlopen) ... ok
test_bad_address (test.test_urllib2_localnet.TestUrlopen) ... skipped "Use of the 'network' resource not enabled"
test_basic (test.test_urllib2_localnet.TestUrlopen) ... ok
test_chunked (test.test_urllib2_localnet.TestUrlopen) ... ok
test_geturl (test.test_urllib2_localnet.TestUrlopen) ... ok
test_https (test.test_urllib2_localnet.TestUrlopen) ... Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:600)
stopping HTTPS server
joining HTTPS thread
ERROR
test_https_sni (test.test_urllib2_localnet.TestUrlopen) ... Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:600)
stopping HTTPS server
joining HTTPS thread
ERROR
test_https_with_cadefault (test.test_urllib2_localnet.TestUrlopen) ... stopping HTTPS server
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:600)
joining HTTPS thread
ok
test_https_with_cafile (test.test_urllib2_localnet.TestUrlopen) ... Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:600)
stopping HTTPS server
joining HTTPS thread
stopping HTTPS server
joining HTTPS thread
ok
test_info (test.test_urllib2_localnet.TestUrlopen) ... ok
test_iteration (test.test_urllib2_localnet.TestUrlopen) ... ok
test_line_iteration (test.test_urllib2_localnet.TestUrlopen) ... ok
test_redirection (test.test_urllib2_localnet.TestUrlopen) ... ok
test_sending_headers (test.test_urllib2_localnet.TestUrlopen) ... ok

======================================================================
ERROR: test_https (test.test_urllib2_localnet.TestUrlopen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 1182, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1086, in endheaders
    self._send_output(message_body)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 924, in _send_output
    self.send(msg)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 859, in send
    self.connect()
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1230, in connect
    server_hostname=sni_hostname)
  File "/home/benjamin/dev/python/3.4/Lib/ssl.py", line 364, in wrap_socket
    _context=self)
  File "/home/benjamin/dev/python/3.4/Lib/ssl.py", line 584, in __init__
    self.do_handshake()
  File "/home/benjamin/dev/python/3.4/Lib/ssl.py", line 811, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/benjamin/dev/python/3.4/Lib/test/test_urllib2_localnet.py", line 548, in test_https
    data = self.urlopen("https://localhost:%s/bizarre" % handler.port)
  File "/home/benjamin/dev/python/3.4/Lib/test/test_urllib2_localnet.py", line 455, in urlopen
    f = urllib.request.urlopen(url, data, **kwargs)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 481, in _open
    '_open', req)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 1225, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 1184, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)>

======================================================================
ERROR: test_https_sni (test.test_urllib2_localnet.TestUrlopen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 1182, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1086, in endheaders
    self._send_output(message_body)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 924, in _send_output
    self.send(msg)
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 859, in send
    self.connect()
  File "/home/benjamin/dev/python/3.4/Lib/http/client.py", line 1230, in connect
    server_hostname=sni_hostname)
  File "/home/benjamin/dev/python/3.4/Lib/ssl.py", line 364, in wrap_socket
    _context=self)
  File "/home/benjamin/dev/python/3.4/Lib/ssl.py", line 584, in __init__
    self.do_handshake()
  File "/home/benjamin/dev/python/3.4/Lib/ssl.py", line 811, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/benjamin/dev/python/3.4/Lib/test/test_urllib2_localnet.py", line 587, in test_https_sni
    self.urlopen("https://localhost:%s" % handler.port)
  File "/home/benjamin/dev/python/3.4/Lib/test/test_urllib2_localnet.py", line 455, in urlopen
    f = urllib.request.urlopen(url, data, **kwargs)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 481, in _open
    '_open', req)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 1225, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/home/benjamin/dev/python/3.4/Lib/urllib/request.py", line 1184, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)>

----------------------------------------------------------------------
Ran 22 tests in 3.087s
msg230542 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-11-03 15:28
Latest patch fixes the urllib2_localnet tests.
msg230548 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-11-03 17:50
Fix for the failing test_ssl testes.
msg230554 - (view) Author: Roundup Robot (python-dev) Date: 2014-11-03 19:37
New changeset 2afe5413d7af by Benjamin Peterson in branch '3.4':
PEP 476: enable HTTPS certificate verification by default (#22417)
https://hg.python.org/cpython/rev/2afe5413d7af

New changeset 731375f83406 by Benjamin Peterson in branch 'default':
merge 3.4 (#22417)
https://hg.python.org/cpython/rev/731375f83406
msg230556 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-11-03 19:39
Okay, 3.4/3.5 have been dealt with. I had to hack up test_logging a bit. (#22788 would make that better). 2.7 now needs a backport.
msg230568 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-11-03 22:41
Somehow the Windows bots are failing to verify python.org http://buildbot.python.org/all/builders/x86%20XP-4%203.x/builds/11179/steps/test/logs/stdio
msg231140 - (view) Author: koobs (koobs) Date: 2014-11-13 23:20
Builds failing on koobs-freebsd9 buildbot for:

3.x: since revision b2c17681404f80edae2ee4846db701104d942cc4
3.4: since revision 246c9570a75798a4757001620cf92cc8d2eba684 	

Attaching both initial build failure test logs.
msg231583 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-11-23 23:30
Attached patch backports this to 2.7.
msg231592 - (view) Author: Roundup Robot (python-dev) Date: 2014-11-24 03:02
New changeset fb83916c3ea1 by Benjamin Peterson in branch '2.7':
pep 476: verify certificates by default (#22417)
https://hg.python.org/cpython/rev/fb83916c3ea1
History
Date User Action Args
2014-11-24 04:58:27benjamin.petersonsetstatus: open -> closed
resolution: fixed
2014-11-24 03:02:23python-devsetmessages: + msg231592
2014-11-23 23:30:09alexsetfiles: + issue22417.diff

messages: + msg231583
2014-11-13 23:20:23koobssetfiles: + koobs-freebsd9.python3x-build2357.log
nosy: + koobs
messages: + msg231140

2014-11-03 22:41:25benjamin.petersonsetmessages: + msg230568
2014-11-03 19:39:45benjamin.petersonsetmessages: + msg230556
2014-11-03 19:37:21python-devsetnosy: + python-dev
messages: + msg230554
2014-11-03 17:50:52alexsetfiles: + issue22417.diff

messages: + msg230548
2014-11-03 15:28:41alexsetfiles: + issue22417.diff

messages: + msg230542
2014-11-03 15:21:23benjamin.petersonsetmessages: + msg230541
2014-11-02 18:13:27alexsetfiles: + issue22417.diff

messages: + msg230512
2014-11-01 04:18:47alexsetfiles: + issue22417.diff

messages: + msg230428
2014-10-31 01:32:55alexsetfiles: + issue22417.diff

messages: + msg230317
2014-10-30 19:00:28alexsetfiles: + issue22417.diff

messages: + msg230288
2014-10-13 17:27:23alexsetkeywords: + needs review
files: + issue22417.diff
messages: + msg229255
2014-10-13 08:18:40raulcdsetnosy: + raulcd
2014-09-25 23:38:14jceasetnosy: + jcea
2014-09-21 09:02:51Arfreversetnosy: + Arfrever
2014-09-20 17:29:09christian.heimessetnosy: + christian.heimes
2014-09-19 13:38:56ncoghlansetpriority: high -> deferred blocker

dependencies: + urllib.request.urlopen should take a "context" (SSLContext) argument
assignee: alex
versions: + Python 2.7, Python 3.4, Python 3.5
nosy: + benjamin.peterson

messages: + msg227093
2014-09-18 18:06:57jwilksetnosy: + jwilk
2014-09-17 08:57:25orsenthilsetnosy: + orsenthil
2014-09-15 16:39:08alexsetnosy: + dstufft
2014-09-15 12:34:29ncoghlancreate