I'm interested to assist with all back port tickets as soon as my internet connection is fixed. A technician is going to check my line again on Tuesday.

I'm not really interested to assist with backport tickets myself. You may nosy me but I may not care at all :)

Christian, Alex, Donald - is there an in-progress patch for this available at all?

2.7.7rc1 is bearing down on us quite rapidly. Given the likely magnitude of this change, should we consider targeting 2.7.8, and give the OpenStack CI team and others a greater chance to check for any issues based on a source checkout before we publish it for general consumption?

I have not started on this patch yet, I'd wanted to get the other, smaller, backports done first. Unless someone else is planning on getting to this, I think 2.7.8 is more realistic.

I agree that 2.7.8 is more realistic. This particular backport is going to be the most work due to the scope of it.

2.7.8 will likely be earlier than expected in order to address the latest OpenSSL update for the Windows installers. So while the likely time frame for this hasn't changed (i.e. November'ish 2014), that release is now expected to be 2.7.9 (assuming the OpenSSL review doesn't find any more surprises, which is a big assumption).

On 20.06.2014 09:26, Nick Coghlan wrote:
> 
> 2.7.8 will likely be earlier than expected in order to address the latest OpenSSL update for the Windows installers. So while the likely time frame for this hasn't changed (i.e. November'ish 2014), that release is now expected to be 2.7.9 (assuming the OpenSSL review doesn't find any more surprises, which is a big assumption).

I think we need to be more careful about using those patch level release
numbers. If we do a new release every time OpenSSL needs to get patched,
we'd probably hit the 2.7.10 wall later this year.

IMO, now would be a good time to discuss how we should deal with
the patch level number turning two digit or preventing that
using some other approach.

I just wanted to note that I've been actively working on this, but it's being difficult in ways I hadn't predicted :-) Will send an update to python-dev in the next week or so.

MAL - agreed on the version numbering implications of treating OpenSSL CVE's as CPython CVE's, but I think Guido pretty much answered that when he extended the 2.7 EOL to 2020 (i.e. we were going to hit 2.7.10 within the next couple of years regardless).

Starting a python-dev thread on that topic in order to reach a broader audience is still a reasonable idea, though.

On 20.06.2014 16:38, Nick Coghlan wrote:
> 
> Nick Coghlan added the comment:
> 
> MAL - agreed on the version numbering implications of treating OpenSSL CVE's as CPython CVE's, but I think Guido pretty much answered that when he extended the 2.7 EOL to 2020 (i.e. we were going to hit 2.7.10 within the next couple of years regardless).
> 
> Starting a python-dev thread on that topic in order to reach a broader audience is still a reasonable idea, though.

Done.

The attached patch (drafted by myself, and David Reid) backports all of the SSL module (and tests!!!) to Python 2.7. All tests pass on my machine (OS X 10.9), I haven't tested against other platforms.

I /suspect/ the best way to review this patch will be by looking at the diff between this patch, and the master branch, in the Lib/ssl.py, Doc/Library/ssl.html, and Modules/_ssl.c files, but of course feel free to review however you like :-)

You can see the complete branch history at: https://github.com/alex/cpython/commits/backport-ssl

For this to work you must apply the patch from http://bugs.python.org/issue22023 first.

New version of this patch fixes a bunch of versionadded and changeds in the docs that referred to the wrong version.

Try #3: This fixes a bug with NPN (I hadn't tested with an OpenSSL that supports NPN, I have now), and cherry-picks the typo fixes from 967311e6c0d2

Awesome! :)

I'll try to find some time to check your work when I'm back from EuroPython.

Thanks for working through this!

I've kicked it in the direction of the Fedora Python SIG folks (https://lists.fedoraproject.org/pipermail/python-devel/2014-July/000611.html), since it would be good if we could get it tested before it makes its way into an upstream release.

New patch cherry pick's the fix from issue22074.

Some results from a quick build/test on OS X 10.9.4 with Xcode 5.1.1 clang and openssl 1.0.1h from MacPorts (similar results when built with Apple-supplied openssl 0.9.8y and on OS X 10.5.8 ppc with Apple-supplied openssl 0.9.7l):

- Modules/_ssl.c:2861:52: warning: invalid conversion specifier 'R'
      [-Wformat-invalid-specifier]
                     "unknown elliptic curve name %R", name);
                                                  ~^
- segfault on test_load_cert_chain (lldb traceback in attached file)

./python -m test.regrtest -v -uall,-largefile test_ssl
== CPython 2.7.8+ (default, Jul 26 2014, 23:44:13) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
==   Darwin-13.3.0-x86_64-i386-64bit little-endian
==   /py/dev/27/source/build/test_python_22427
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
test_ssl
test_ssl: testing with 'OpenSSL 1.0.1h 5 Jun 2014' (1, 0, 1, 8, 15)
          under Mac ('10.9.4', ('', '', ''), 'x86_64')
          HAS_SNI = True
          OP_ALL = 0x800003ff
          OP_NO_TLSv1_1 = 0x10000000
test__create_stdlib_context (test.test_ssl.ContextTests) ... ok
test_cert_store_stats (test.test_ssl.ContextTests) ... ok
test_check_hostname (test.test_ssl.ContextTests) ... ok
test_ciphers (test.test_ssl.ContextTests) ... ok
test_constructor (test.test_ssl.ContextTests) ... ok
test_create_default_context (test.test_ssl.ContextTests) ... ok
test_get_ca_certs (test.test_ssl.ContextTests) ... ok
test_load_cert_chain (test.test_ssl.ContextTests) ... Segmentation fault: 11

- test_poplib hangs, requiring ctrl-c interrupt

/python -m test.regrtest -v -uall test_poplib
== CPython 2.7.8+ (default, Jul 26 2014, 23:44:13) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
==   Darwin-13.3.0-x86_64-i386-64bit little-endian
==   /py/dev/27/source/build/test_python_33661
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
test_poplib
test_dele (test.test_poplib.TestPOP3Class) ... ok
test_exceptions (test.test_poplib.TestPOP3Class) ... ok
test_getwelcome (test.test_poplib.TestPOP3Class) ... ok
test_list (test.test_poplib.TestPOP3Class) ... ok
test_noop (test.test_poplib.TestPOP3Class) ... ok
test_pass_ (test.test_poplib.TestPOP3Class) ... ok
test_retr (test.test_poplib.TestPOP3Class) ... ok
test_rpop (test.test_poplib.TestPOP3Class) ... ok
test_stat (test.test_poplib.TestPOP3Class) ... ok
test_top (test.test_poplib.TestPOP3Class) ... ok
test_uidl (test.test_poplib.TestPOP3Class) ... ok
test_user (test.test_poplib.TestPOP3Class) ... ok
testTimeoutDefault (test.test_poplib.TestTimeouts) ... ok
testTimeoutNone (test.test_poplib.TestTimeouts) ... ok
testTimeoutValue (test.test_poplib.TestTimeouts) ... ok
test__all__ (test.test_poplib.TestPOP3_SSLClass) ... Exception in thread Thread-16:
Traceback (most recent call last):
  File "/py/dev/27/source/Lib/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/py/dev/27/source/Lib/test/test_poplib.py", line 132, in run
    asyncore.loop(timeout=0.1, count=1)
  File "/py/dev/27/source/Lib/asyncore.py", line 220, in loop
    poll_fun(timeout, map)
  File "/py/dev/27/source/Lib/asyncore.py", line 156, in poll
    read(obj)
  File "/py/dev/27/source/Lib/asyncore.py", line 87, in read
    obj.handle_error()
  File "/py/dev/27/source/Lib/asyncore.py", line 83, in read
    obj.handle_read_event()
  File "/py/dev/27/source/Lib/asyncore.py", line 443, in handle_read_event
    self.handle_accept()
  File "/py/dev/27/source/Lib/test/test_poplib.py", line 143, in handle_accept
    self.handler = self.handler(conn)
  File "/py/dev/27/source/Lib/test/test_poplib.py", line 242, in __init__
    self.push('+OK dummy pop3 server ready.')
  File "/py/dev/27/source/Lib/test/test_poplib.py", line 63, in push
    asynchat.async_chat.push(self, data + '\r\n')
  File "/py/dev/27/source/Lib/asynchat.py", line 193, in push
    self.initiate_send()
  File "/py/dev/27/source/Lib/asynchat.py", line 244, in initiate_send
    self.handle_error()
  File "/py/dev/27/source/Lib/asynchat.py", line 242, in initiate_send
    num_sent = self.send(data)
  File "/py/dev/27/source/Lib/asyncore.py", line 374, in send
    result = self.socket.send(data)
  File "/py/dev/27/source/Lib/ssl.py", line 662, in send
    return self._sslobj.write(data)
SSLWantReadError: The operation did not complete (read) (_ssl.c:1611)

^CWarning -- asyncore.socket_map was modified by test_poplib

Test suite interrupted by signal SIGINT.
1 test omitted:
    test_poplib
[51620 refs]

- new test/capath directory needs to be added to LIBSUBDIRS in Makefile.pre.in

Ned, did you apply the patch from http://bugs.python.org/issue22023 first? That bt looks like what you'd see without it.

Does https://github.com/alex/cpython/commits/backport-ssl contain all relevant code changes? I like to review the changes during my train ride home.

> did you apply the patch from http://bugs.python.org/issue22023 first?

Sorry, missed that.  Let's try again with that applied as well:

OS X 10.9.4 with Xcode 5.1.1 clang and openssl 1.0.1h from MacPorts

- Modules/_ssl.c:2861:52: warning: invalid conversion specifier 'R' (see above)

- test_poplib hangs, requiring ctrl-c interrupt (see above)

- certificate verify failed in test_algorithms of test_ssl

test_ssl: testing with 'OpenSSL 1.0.1h 5 Jun 2014' (1, 0, 1, 8, 15)
          under Mac ('10.9.4', ('', '', ''), 'x86_64')
          HAS_SNI = True
          OP_ALL = 0x800003ff
          OP_NO_TLSv1_1 = 0x10000000
[...]
======================================================================
ERROR: test_algorithms (test.test_ssl.NetworkedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/27/root/uxd_macports/lib/python2.7/test/test_ssl.py", line 1475, in test_algorithms
    s.connect(remote)
  File "/py/dev/27/root/uxd_macports/lib/python2.7/ssl.py", line 810, in connect
    self._real_connect(addr, False)
  File "/py/dev/27/root/uxd_macports/lib/python2.7/ssl.py", line 801, in _real_connect
    self.do_handshake()
  File "/py/dev/27/root/uxd_macports/lib/python2.7/ssl.py", line 774, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)

----------------------------------------------------------------------
Ran 99 tests in 13.035s

FAILED (errors=1, skipped=4)

when built with Apple-supplied openssl 0.9.8y on OS X 10.9.4:

- test_algorithms does not fail:

test_ssl: testing with 'OpenSSL 0.9.8y 5 Feb 2013' (0, 9, 8, 25, 15)
          under Mac ('10.9.4', ('', '', ''), 'x86_64')
          HAS_SNI = True
          OP_ALL = 0x     7ff
[...]
test_algorithms (test.test_ssl.NetworkedTests) ...
Cipher with ('sha256.tbs-internet.com', 443) is ('RC4-SHA', 'TLSv1/SSLv3', 128)
Certificate is:
{'OCSP': (u'http://ocsp.tbs-x509.com',),
 'caIssuers': (u'http://crt.tbs-internet.com/TBSX509CASGC.crt',
               u'http://crt.tbs-x509.com/TBSX509CASGC.crt'),
 'crlDistributionPoints': (u'http://crl.tbs-internet.com/TBSX509CASGC.crl',
                           u'http://crl.tbs-x509.com/TBSX509CASGC.crl'),
[...]

- a non-fatal broken pipe exception is seen

test_nonblocking_send (test.test_ssl.ThreadedTests) ...  server:  new connection from ('127.0.0.1', 50583)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
 server: selected protocol is now None
Exception in thread Thread-36:
Traceback (most recent call last):
  File "/py/dev/27/root/uxd/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/py/dev/27/root/uxd/lib/python2.7/test/test_ssl.py", line 1636, in run
    self.write(msg.lower())
  File "/py/dev/27/root/uxd/lib/python2.7/test/test_ssl.py", line 1581, in write
    return self.sslconn.write(bytes)
  File "/py/dev/27/root/uxd/lib/python2.7/ssl.py", line 622, in write
    return self._sslobj.write(data)
error: [Errno 32] Broken pipe

ok

on OS X 10.5.8 ppc with Apple-supplied openssl 0.9.7l:

- test_rude_shutdown fails

test_ssl: testing with 'OpenSSL 0.9.7l 28 Sep 2006' (0, 9, 7, 12, 15)
          under Mac ('10.5.8', ('', '', ''), 'PowerPC')
          HAS_SNI = False
          OP_ALL = 0x     7ff
[...]
======================================================================
ERROR: test_rude_shutdown (test.test_ssl.ThreadedTests)
A brutal shutdown of an SSL server should raise an OSError
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/test/test_ssl.py", line 2139, in test_rude_shutdown
    connector()
  File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/test/test_ssl.py", line 2130, in connector
    ssl_sock = ssl.wrap_socket(c)
  File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/ssl.py", line 868, in wrap_socket
    ciphers=ciphers)
  File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/ssl.py", line 561, in __init__
    self.do_handshake()
  File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/ssl.py", line 774, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 54] Connection reset by peer

----------------------------------------------------------------------
Ran 99 tests in 14.843s

FAILED (errors=1, skipped=21)

- new test/capath directory needs to be added to LIBSUBDIRS in Makefile.pre.in

Investigations:

The test_poplib failures are caused by the backporting of http://bugs.python.org/issue20951, does anyone have opinions on whether we should backport Antoine's fix for poplib, or revert that particular change?

I'm not sure how to fix that compilation warning, any suggestions?

I've uploaded a new patch which fixes the test_algorithm's failure with non-system OS X.

Latest patch makes the needed change to Makefile.pre.in

Would it be possible to also backport the changes to httplib.py to support TLS SNI as many libraries (including setuptools) relies on this module to download files and some servers reject clients not supporting TLS SNI (such as cloud.github.com)? I have added a patch for Python 2.7.8. Thanks for working on the backports of SSL changes!

Hi Arnaud, such a change is out of scope for PEP 466. To minimise the scale of the changes in a maintenance release, it was deliberately limited to provide the core infrastructure, so at least third party libraries like requests can support SNI in Python 2, without touching every networking module in the Python 2 standard library.

However, you're welcome to write a follow up PEP suggesting expanding the scope of the changes out to at least offering SNI support in other standard library modules. While I can't promise such a proposal would be approved, I do agree the question is worth asking. The risk of that change can then be independently assessed vs advising users that need SNI support in Python 2 to install a third party module like requests. (The bootstrapping problem is likely to be addressed by backporting the relevant parts of PEP 453, although that project too is in need of a volunteer to write the enhancement proposal).

I think we probably want to revert that particular change. Afaik it wasn't added to 3.4 because of the danger of breaking things so we probably shouldn't add it to 2.7.

Latest patch does as Donald suggests and rolls back the WantWrite changes, fixing poplib.

To help keep the file list clear, I unlinked the older versions of the patch. They're still available in the issue history below.

Alex, could you generate a version using the hg diff format (so the review gets generated automatically), or else add it manually to Rietveld?

I've been doing this work in a git repo, do you know how to generate an hg-formated diff with git?

Doesn't hg accept git diffs?

On Wed, Aug 6, 2014, at 21:31, Alex Gaynor wrote:
> 
> Alex Gaynor added the comment:
> 
> I've been doing this work in a git repo, do you know how to generate an
> hg-formated diff with git?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21308>
> _______________________________________

The problem is that the Rietveld integration isn't able to figure out the base revision, so it isn't generating the review automatically. Times like this it would be nice to be able to specify a "base branch" explicitly...

In other news... the outcome of the Fedora thread is that Slavek is just going to go ahead and apply the patch to Python 2.7 RPMs in Fedora Rawhide next week. That will hopefully be entirely uneventful, but if it isn't, at least we'll know *before* merging it upstream :)

Le 07/08/2014 04:34, Nick Coghlan a écrit :
>
> In other news... the outcome of the Fedora thread is that Slavek is
just going to go ahead and apply the patch to Python 2.7 RPMs in Fedora
Rawhide next week. That will hopefully be entirely uneventful, but if it
isn't, at least we'll know *before* merging it upstream :)

I doubt adding a ton of new APIs and code can be uneventful, but good 
luck :)

New patch should be in the "mercurial" diff format.

On 7 Aug 2014 23:11, "Antoine Pitrou" <report@bugs.python.org> wrote:
> I doubt adding a ton of new APIs and code can be uneventful, but good
> luck :)

They don't call it "Rawhide" for nothing! :)

Hi everyone,

I went ahead and I've applied ssl-backport.diff (alex, 2014-08-07 18:49) patch into Python 2.7.8 on Fedora Rawhide (currently only scratch build).

My report:
Firstly, I've encountered seg fault, I fixed this with patch from http://bugs.python.org/issue22023

Next issue was/is distro specific, there is disabled SSLv2 protocol when SSLv23_method is used[0]. However python3 builds fine[1]:

In test_ssl.py:test_protocol_sslv23 when calling
try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_SSLv2, True)
this will raise an OSError on python3 but this error is caught by test.

In python 2 this will raise a socket.error[2] instead, but test_protocol_sslv23 is catching ssl.SSLError, shouldn't be that changed to socket.error so both python2 and python3 behave the same? 


[0] SSLv23 http://pkgs.fedoraproject.org/cgit/openssl.git/tree/openssl-1.0.1h-disable-sslv2v3.patch
[1] Python3 build https://kojipkgs.fedoraproject.org//work/tasks/9993/7399993/build.log (search for test_protocol_sslv23)
[2] Python2 build https://kojipkgs.fedoraproject.org//work/tasks/9367/7399367/build.log (search for test_protocol_sslv23)

Thanks for testing this out Robert! The attached patch fixes the error you saw.

New patch additionally backports the Tools/ssl/ directory from Python3, which has two utilities for generating some of the code used. Thanks to Benjamin for catching this.

Last version had some stray stuff due to the fact that I don't know how to use version control. New version should resolve that.

Running test_ssl gives this:

% ./python Lib/test/regrtest.py test_ssl    
test_ssl
Exception in thread Thread-104:
Traceback (most recent call last):
  File "/home/benjamin/dev/python/2.7/Lib/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/benjamin/dev/python/2.7/Lib/test/test_ssl.py", line 1594, in run
    if not self.wrap_conn():
  File "/home/benjamin/dev/python/2.7/Lib/test/test_ssl.py", line 1545, in wrap_conn
    self.sock, server_side=True)
  File "/home/benjamin/dev/python/2.7/Lib/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/home/benjamin/dev/python/2.7/Lib/ssl.py", line 561, in __init__
    self.do_handshake()
  File "/home/benjamin/dev/python/2.7/Lib/ssl.py", line 784, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 104] Connection reset by peer

On what platform?

Gentoo Linux. Openssl 1.0.1i

On Tue, Aug 19, 2014, at 19:29, Alex Gaynor wrote:
> 
> Alex Gaynor added the comment:
> 
> On what platform?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21308>
> _______________________________________

In load_cert_chain, you may find it convenient to not Py_DECREF and Py_CLEAR keyfile_bytes and certfile_bytes, which aren't PyObject *.

Latest patch fixes both the issues Benjamin noted.

I spent hours looking at this patch, which certainly doesn't constitute a real review, but is probably about as good as your going to get on this behemouth. Anyway, Alex knows he's on the hook for when things start going sour.

New changeset 221a1f9155e2 by Benjamin Peterson in branch '2.7':
backport many ssl features from Python 3 (closes #21308)
http://hg.python.org/cpython/rev/221a1f9155e2

The current implementation doesn't work with Unicode file paths. Try passing a Unicode string e.g. as "cafile" into context.load_verify_locations(). It calls PyString_AsEncodedObject() on it, which then fails with a PyErr_BadArgument() on the entry type check.

Thanks for the report, I've filed: http://bugs.python.org/issue22244 to track that issue.

Thanks for getting this in, folks!

As Alex suggests, we can file any identified regressions as new issues.

Note that this change broke eventlet: https://github.com/eventlet/eventlet/issues/135

Filed http://bugs.python.org/issue22438 to track it.

causing #22523, still referencing _ssl.sslwrap.