https connections fails under intel MAC OS X 10.6.6 and 10.6.7 using httplib and/or urllib2 connecting to an IIS web server  requesting basic authentication and a client certificate.

This is an issue with MAC OS X 10.6.x, because I tried these very same  scripts under RHES 5 x64 with python 2.6.5 and python 2.7.1 and they both run just fine.


----------------------------

Sample code with urllib2:

import urllib2
response = urllib2.urlopen('https://www.finratrace.org/')

------------------
Sample code with httplib

import httplib,base64

key,cert=('/tmp/cert_nopwd.pem',)*2
HOSTNAME = 'www.finratrace.org'

username,password='myuser','mypass'

base64string = base64.encodestring('%s:%s' % (username, password))[:-1]


conn = httplib.HTTPSConnection(HOSTNAME,key_file = key,cert_file = cert)
conn.putrequest('GET', '/')
conn.putheader("Authorization", "Basic %s" % base64string)

conn.endheaders()
response = conn.getresponse()

-----------------------

Both samples send the following error:

====>With python 2.6.1

Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> response = urllib2.urlopen('https://www.finratrace.org/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 383, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 401, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 361, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 1138, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 1103, in do_open
    r = h.getresponse()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 950, in getresponse
    response.begin()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 390, in begin
    version, status, reason = self._read_status()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 354, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine

==================================
====>Same thing happens Using python 2.7.1

Python 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib,base64
>>> 
>>> key,cert=('/tmp/cert_nopwd.pem',)*2
>>> HOSTNAME = 'www.finratrace.org'
>>> 
>>> username,password='myuser','mypass'
>>> 
>>> base64string = base64.encodestring('%s:%s' % (username, password))[:-1]
>>> 
>>> 
>>> conn = httplib.HTTPSConnection(HOSTNAME,key_file = key,cert_file = cert)
>>> conn.putrequest('GET', '/')
>>> conn.putheader("Authorization", "Basic %s" % base64string)
>>> 
>>> conn.endheaders()
>>> response = conn.getresponse()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1021, in getresponse
    return response
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 401, in begin
    while True:
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 365, in _read_status
    # sending a valid response.
httplib.BadStatusLine: ''

=====================================

I also compiled py 2.7 getting the same error message.

I tried patches from Issue7291, but did not work.

The site you mention requires client certificates to log on. 

On my machine (OSX 10.6) urllib2 example just hangs. I haven't checked yet why this happens, or if this also happens on other OS-es or releases.

Other HTTPS sites work fine for me though.

Are you sure that the RHES machine doesn't have a special configuration that provides the client certificates?

Yep, I am sure Ronald, the RH server has no special configuration. 

The RH box actually immediately connects to the server, an because it requires a certificate returns a 403 forbidden code:


[opentrails@redhat5 ~]$ uname -a
Linux redhat5.ultralat.com 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
[opentrails@redhat5 ~]$ python
Python 2.6.5 (r265:79063, Dec  1 2010, 19:40:01) 
[GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> response = urllib2.urlopen('https://www.finratrace.org/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/local/lib/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/local/lib/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

--------------------------------------------------------------

And "HTTP Error 403: Forbidden: is what MAC OS X should return as well when the certificate was not sent.

==============================================================

Furthermore, MAC OS X hangs even with the second sample code, which provides the certificate and the authentication, failing as well.

See the section "====>Same thing happens Using python 2.7.1" on my original post. That example sends the certificate and the authentication but it simply hangs, getting the "httplib.BadStatusLine: ''" error at the end.

--------------------------------------------------------------
Now, if I send the authentication and certificate under the linux box, it works fine:



[opentrails@redhat5 ~]$ uname -a
Linux redhat5.ultralat.com 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
[opentrails@redhat5 ~]$ python
Python 2.6.5 (r265:79063, Dec  1 2010, 19:40:01) 
[GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib,base64
>>> 
>>> key,cert=('/tmp/cert_nopwd.pem',)*2
>>> HOSTNAME = 'www.finratrace.org'
>>> 
>>> username,password='mysuer','mypass'
>>> 
>>> base64string = base64.encodestring('%s:%s' % (username, password))[:-1]
>>> 
>>> 
>>> conn = httplib.HTTPSConnection(HOSTNAME,key_file = key,cert_file = cert)
>>> conn.putrequest('GET', '/')
>>> conn.putheader("Authorization", "Basic %s" % base64string)
>>> 
>>> conn.endheaders()
>>> response = conn.getresponse()
>>> print response.status,response.reason
200 OK
>>> html= response.read()
>>> open("/tmp/kk.html",'w').write(html)
>>> 

I attached the response file from the web site.

---------------------------------------------
By the way, I am using openssl 0.9.8 in both machines:

imac:bin isaac$ uname -a 
Darwin imac.localdomain 10.7.0 Darwin Kernel Version 10.7.0: Sat Jan 29 15:16:10 PST 2011; root:xnu-1504.9.37~1/RELEASE_X86_64 x86_64
imac:bin isaac$ openssl version
OpenSSL 0.9.8l 5 Nov 2009


- - - - - - - - - - - - - -

Redhat has two  openssl flavors:

[opentrails@redhat5 ~]$ openssl version
OpenSSL 0.9.8b 04 May 2006
[opentrails@redhat5 ~]$ /usr/local/ssl/bin/openssl version
OpenSSL 0.9.8p 16 Nov 2010

-------------------------------------------------------------

In case it might help, this was working with MAC OS X 10.4 with PPC. It started to fail when I upgraded to an intel iMac with OSX 10.6. I do not know if it has something to do with the 64 bits kernel.

This appears to be a bug in the versions of the Apple-supplied openssl libs supplied in OS X 10.5 (0.9.7l) and 10.6 (0.9.8l).  You can see the same results using the openssl test client:

   $ openssl s_client -connect www.finratrace.org:443

With the Apple-supplied versions in 10.5 and 10.6, it hangs. With a newer MacPorts-installed openssl (I have a 1.0.0d), the openssl s_client exits after the error. I verified that the urllib.request in the MacPorts python3.2 which is linked with the newer openssl also correctly does not hang.

I have filed issue11736 as a more or less related (or bogus) issue.

Ned: we could work around this platform issue by including openssl 1.0d in the OSX installer.  I'm not sure if that's acceptable for a bugfix release though.

The output for the command: 

   $ openssl s_client -connect www.finratrace.org:443

was the same on MAC OS X 10.6 and on Red hat 5 (https works fine under linux).

Nevertheless, Ned Deily is right: the bug is on the openssl libs supplied with OS X 10.6 

To solve this issue, I compiled and install OpenSSL 1.0.0d and then link python against this library.

This is the full recipe, step by step:

1. Install openssl. 
   Download the source tar for openssl. I used version openssl-1.0.0d.
   To build 64-bit library, then you have to invoke 
   './Configure darwin64-x86_64-cc' *manually*. Also, to make ssl work in python,
   the openssl libraries must be 'shared libraries'. 
   

   First, Expand the tar file into a temporary directory, I am using /tmp:
   
   $ cd /tmp
   
   $ tar xvzf openssl-1.0.0d.tar.gz
   
   $ cd openssl-1.0.0d
   

   To Build openssl as 64 bits shared libraries and install it:
   
   $ ./Configure darwin64-x86_64-cc shared
   
   $ make 
   
   $ make test # this step is optional
   
   $ sudo make install
   
   
   This will install openssl in the default directory: /usr/local/ssl
   
2. Compile and install python. 
   Download the source tar file. I used version Python 2.7.2
   
   a) Expand the tar file (again into a temporary directory)
   
   b) then go into the Modules folder
   
   c) vi the Setup.dist file, looking for the SSL string (if your are not familiar 
      with vi, you can use any text editor), then uncomment the lines BELLOW the 
      message:
      
               "# Socket module helper for SSL support ..."
               
      Your file must look as follows:
   
   
# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
SSL=/usr/local/ssl
_ssl _ssl.c \
        -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
        -L$(SSL)/lib -lssl -lcrypto
 
   

   d) Then using python defaults (installing under /usr/local) execute:
   
   $   ./configure 

   $ make

   $ make test # optional

   $ sudo make install
   

3. To test if python now has ssl support, start python and execute
   these commands (be sure you invoke the new python under /usr/local/bin):

imac:~ isaac$ /usr/local/bin/python
Python 2.7.2 (default, Jun 30 2011, 16:00:06) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> hasattr(httplib, 'HTTPS')
True
>>> # MUST be  True, otherwise has NO ssl support
... 
>>> import socket
>>> hasattr(socket,'ssl')
True
>>> # MUST be  True, otherwise has NO ssl support
... 
>>> import _ssl
>>> # should NOT give any error when importing
... 
>>> 


That's all, now you have ssl support with python under MAC OS X 10.6