classification
Title: HTTPS/TLS Problem in Python 3.3
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.3, Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, jcea, loewis, neologix, pitrou, pventura
Priority: normal Keywords:

Created on 2012-10-29 22:52 by pventura, last changed 2015-02-16 13:04 by loewis. This issue is now closed.

Files
File name Uploaded Description Edit
demo.py pventura, 2012-10-29 22:52 Demo program showing the behavior
py2-filtered.pcapng pventura, 2012-10-29 22:53 Wireshark capture file for Python 2.7.3
py3-filtered.pcapng pventura, 2012-10-29 22:53 Wireshark capture file for Python 3.3.0
linux-tls10-handshake.txt pitrou, 2012-10-30 19:18
windows-tls12-handshake.txt pitrou, 2012-10-30 19:18
Messages (8)
msg174158 - (view) Author: Phil (pventura) Date: 2012-10-29 22:52
I had converted some code for a scraper from 2.7.3 to 3.3.0 on Windows 7 and suddenly the code stopped working.  Now the https fetch results in:
Traceback (most recent call last):
  File "D:\Users\Phil\Desktop\demo.py", line 67, in <module>
    page=getWebData()
  File "D:\Users\Phil\Desktop\demo.py", line 59, in getWebData
    response=urllib.request.urlopen(req, cadefault=False)
  File "D:\Program Files\Python33\lib\urllib\request.py", line 160, in urlopen
    return opener.open(url, data, timeout)
  File "D:\Program Files\Python33\lib\urllib\request.py", line 473, in open
    response = self._open(req, data)
  File "D:\Program Files\Python33\lib\urllib\request.py", line 491, in _open
    '_open', req)
  File "D:\Program Files\Python33\lib\urllib\request.py", line 451, in _call_chain
    result = func(*args)
  File "D:\Program Files\Python33\lib\urllib\request.py", line 1287, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "D:\Program Files\Python33\lib\urllib\request.py", line 1255, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>
 
I have run Wireshark using Python 2.7.3 and Python 3.3.0 (see attached files)
msg174159 - (view) Author: Phil (pventura) Date: 2012-10-29 22:55
You will notice that in the Python 3.3.0 version after packet 54 (Client Hello), there seems to be no response from the server.
Today, I was able to verify that the code worked under Python 3.2.3.
msg174186 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-10-30 09:23
Well, the first difference that jumps out is that with python 2.7, the protocol used is SSLv2, whereas it's bare SSL on Python 3.3.0.

But another interesting thing is the presence, in Python 2.3, of many extenstions (elliptic_curves, heartbeat, server_name, etc).

It's likely that the server doesn't support one of these extensions (but I guess it should answer accordingly instead of ignoring the hello).
msg174214 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-30 18:52
Your script works for me under Linux with Python 3.2, 3.3 and 3.4.
Perhaps the problem has to do with the version of OpenSSL that we package Windows binaries with? My OpenSSL version here (as given by ssl.OPENSSL_VERSION) is 'OpenSSL 1.0.0d 8 Feb 2011'.
msg174216 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-30 19:18
It looks like we use OpenSSL 1.0.1c for Python 3.3 Windows builds, which would explain the different Client Hello. In your capture, the client identifies itself as TLS 1.2, while on my Linux box it's TLS 1.0.

I'm uploading the decoded protocol tree of the two handshakes.
msg174219 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-30 19:44
Note that you could work around the issue by forcing a lower SSL version:

    ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv3)
    opener = urllib.request.build_opener(
        urllib.request.HTTPCookieProcessor(cj),                                    
        urllib.request.HTTPSHandler(context=ssl_context))
    urllib.request.install_opener(opener)

(I would suggest doing this only for this particular HTTPS site, not globally, since using a lower SSL version may make the exchanges potentially less secure)
msg236027 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-02-15 07:54
On Windows 3.4.2 ssl.OPENSSL_VERSION is 'OpenSSL 1.0.1i 6 Aug 2014' and 3.5 is currently being built with 1.0.1l so is there anything that our windows developers need to do here with 3.3?
msg236102 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2015-02-16 13:04
Python 3.3 no longer receives Windows releases, so closing this issue as out of date. Phil, if the issue persists in a newer Python version, please re-report.
History
Date User Action Args
2015-02-16 13:04:44loewissetstatus: open -> closed
resolution: out of date
messages: + msg236102
2015-02-15 07:54:58BreamoreBoysetnosy: + BreamoreBoy
messages: + msg236027
2014-02-15 00:33:59jceasetnosy: + jcea
2012-10-30 19:44:25pitrousetmessages: + msg174219
2012-10-30 19:18:24pitrousetfiles: + windows-tls12-handshake.txt
2012-10-30 19:18:17pitrousetfiles: + linux-tls10-handshake.txt
2012-10-30 19:18:10pitrousetmessages: + msg174216
2012-10-30 18:52:15pitrousetversions: + Python 3.4
nosy: + loewis

messages: + msg174214

components: + Windows
2012-10-30 09:23:11neologixsetnosy: + pitrou, neologix
messages: + msg174186
2012-10-29 22:55:33pventurasetmessages: + msg174159
2012-10-29 22:53:36pventurasetfiles: + py3-filtered.pcapng
2012-10-29 22:53:11pventurasetfiles: + py2-filtered.pcapng
2012-10-29 22:52:02pventuracreate