➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: urllib2/httplib is rendering 400s for every authenticated-SSL request, suddenly
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Dustin.Oprea, dsoprea, pitrou, r.david.murray
Priority: normal Keywords:

Created on 2014-11-10 05:57 by Dustin.Oprea, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (8)
msg230934 - (view) Author: Dustin Oprea (Dustin.Oprea) Date: 2014-11-10 05:57 
I am trying to do an authenticated-SSL request to an Nginx server using *requests*, which wraps urllib2/httplib. It's worked perfectly for months until Friday on my local system (Mac 10.9.5), and there have been no upgrades/patches. 

My Python 2.7.6 client fails when connecting to Nginx, locally. I get a 400, with this:

<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

This is an example that uses urllib2/httplib, directly:

import urllib2
import httplib

cert_filepath = '/var/lib/rt_data/ssl/rt.crt.pem'
key_filepath = '/var/lib/rt_data/ssl/rt.private_key.pem'

url = 'https://deploy_api.local:8443/auth/admin/1/hosts'


class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
    """Wrapper to allow for authenticated SSL connections."""

    def __init__(self, key, cert):
        urllib2.HTTPSHandler.__init__(self)
        self.key = key
        self.cert = cert

    def https_open(self, req):
        # Rather than pass in a reference to a connection class, we pass in
        # a reference to a function which, for all intents and purposes,
        # will behave as a constructor
        return self.do_open(self.getConnection, req)

    def getConnection(self, host, timeout=300):
        return httplib.HTTPSConnection(host, key_file=self.key, cert_file=self.cert)

opener = urllib2.build_opener(HTTPSClientAuthHandler(key_filepath, cert_filepath))
response = opener.open(url)

response_data = response.read()
print(response_data)

These are the factors:

- It works when connecting to the remote server. Both local and remote are Nginx with similar configs.
- cURL works perfectly:

  curl -s -v -X GET -k --cert /var/lib/rt_data/ssl/rt.crt.pem --key /var/lib/rt_data/ssl/rt.private_key.pem https://server.local:8443/auth/admin/1/hosts

- I've tried under Vagrant with Ubuntu 12.04 (2.7.3) and 14.04 (2.7.6). No difference.
- It works with Python 3.4 on the local system. This only has only affected 2.7 very suddenly.

Due to the error-message above, it seems like there's a break down in sending the certificate/key.

I have no idea what's going on, and this has caused me a fair amount of distress. Can you provide me a direction?
msg230964 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-11-10 15:46 
If you haven't updated Python, then it is hard to see how this could be a python bug.  Not impossible, but you'll have to narrow down the problem before you'd be able to demonstrate it as a python bug, I'm afraid.

If you want help diagnosing this, you might try the python-list mailing list, though it sounds like a nginx forum might be equally useful at this stage (determining what exactly is going on protocol wise that is different between your two test cases).
msg230978 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-11-10 22:21 
No upgrades on the server either?
msg231001 - (view) Author: Dustin Oprea (dsoprea) * Date: 2014-11-11 06:50 
I think I was getting mixed results by using requests and urllib2/3. After nearly being driven crazy, I performed the following steps:

1. Recreated client certificates, and verified that the correct CA was being used from Nginx.

2. Experimenting using an SSL-wrapped client-socket directly, in tandem with s_client.

3. I then removed all of my virtualhosts except for a new one that pointed to a flat directory, just to make sure that I wasn't activating the wrong virtualhost, and there weren't any other complexities.

4. Implemented a bonafide, signed, SSL certificate on my local system, and overriding the hostname using /etc/hosts.

5. This got me past the 400. I switched back to using my local hostname with my self-signed certificate, and told wrap_socket to not verify (at this point, I stopped checking with s_client).

6. I started reactivating all of my normal virtualhost includes, one include at a time.

7. Reverted back to using the standard, proprietary client, and verified that it worked.

I'm guessing that a) something happened to my original certificates, b) I might've had an incorrect CA certificate for authentication, and/or c) I had added a default virtualhost on the non-standard port that I am using that always returns Forbidden, and this might've been unexpectedly catching the wrong requests.

Since I verified my client certificates against my internal issuer in the beginning, I don't think it's (a) or (b).

I could've done without these problems. I can't even say what started it all.
msg231015 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-11-11 09:43 
Le 11/11/2014 07:50, Dustin Oprea a écrit :
> 
> Dustin Oprea added the comment:
> 
> I think I was getting mixed results by using requests and urllib2/3. After nearly being driven crazy, I performed the following steps:
> 
> 1. Recreated client certificates, and verified that the correct CA was being used from Nginx.
> 
> 2. Experimenting using an SSL-wrapped client-socket directly, in tandem with s_client.
> 
> 3. I then removed all of my virtualhosts except for a new one that pointed to a flat directory, just to make sure that I wasn't activating the wrong virtualhost, and there weren't any other complexities.
> 
> 4. Implemented a bonafide, signed, SSL certificate on my local system, and overriding the hostname using /etc/hosts.
> 
> 5. This got me past the 400.

Ah! Perhaps your HTTPS setup was relying on SNI to select the proper
virtual host ?
msg231027 - (view) Author: Dustin Oprea (dsoprea) * Date: 2014-11-11 14:43 
I usually use both on my local system.

Dustin
On Nov 11, 2014 4:43 AM, "Antoine Pitrou" <report@bugs.python.org> wrote:

>
> Antoine Pitrou added the comment:
>
> Le 11/11/2014 07:50, Dustin Oprea a écrit :
> >
> > Dustin Oprea added the comment:
> >
> > I think I was getting mixed results by using requests and urllib2/3.
> After nearly being driven crazy, I performed the following steps:
> >
> > 1. Recreated client certificates, and verified that the correct CA was
> being used from Nginx.
> >
> > 2. Experimenting using an SSL-wrapped client-socket directly, in tandem
> with s_client.
> >
> > 3. I then removed all of my virtualhosts except for a new one that
> pointed to a flat directory, just to make sure that I wasn't activating the
> wrong virtualhost, and there weren't any other complexities.
> >
> > 4. Implemented a bonafide, signed, SSL certificate on my local system,
> and overriding the hostname using /etc/hosts.
> >
> > 5. This got me past the 400.
>
> Ah! Perhaps your HTTPS setup was relying on SNI to select the proper
> virtual host ?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22835>
> _______________________________________
>
msg231042 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-11-11 19:21 
In any case, it sounds like your problem is fixed, so we can close this issue.
msg231053 - (view) Author: Dustin Oprea (Dustin.Oprea) Date: 2014-11-11 21:37 
Agreed. Thank you, @Antoine.

On Tue, Nov 11, 2014 at 2:21 PM, Antoine Pitrou <report@bugs.python.org>
wrote:

>
> Antoine Pitrou added the comment:
>
> In any case, it sounds like your problem is fixed, so we can close this
> issue.
>
> ----------
> resolution:  -> not a bug
> status: open -> closed
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22835>
> _______________________________________
>
History
Date User Action Args
2022-04-11 14:58:10adminsetgithub: 67024
2014-11-11 21:37:17Dustin.Opreasetmessages: + msg231053
2014-11-11 19:21:32pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg231042
2014-11-11 14:43:25dsopreasetmessages: + msg231027
2014-11-11 09:43:26pitrousetmessages: + msg231015
2014-11-11 06:50:53dsopreasetnosy: + dsoprea
messages: + msg231001
2014-11-10 22:21:36pitrousetnosy: + pitrou
messages: + msg230978
2014-11-10 15:46:50r.david.murraysetnosy: + r.david.murray
messages: + msg230964
2014-11-10 05:57:06Dustin.Opreacreate