urllib2 cannot handle https with proxy requiring authorization.

After https_proxy is set correctly,

Python 2.6.4 (r264:75706, Oct 29 2009, 15:38:25)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> c=urllib2.urlopen("https://sourceforge.net")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 389, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 407, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1154, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1121, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error Tunnel connection failed: 407 Proxy
Authentication Required>

This is because HTTPConnection::_tunnel() in httplib.py doesn't send
Proxy-Authorization header.

I created a patch.
I added additional argument 'headers' to HTTPConnection::set_tunnel()
method,
which is a mapping of HTTP headers to sent with CONNECT method. Since
authorization
credential is already set to Request object, in
AbstractHTTPHandler::do_open(),
if "Proxy-Authorization" header is found, pass it to set_tunnel().

It works fine for me.

The patch looks good to me.

IMHO this should be backported to 2.6 as well.

I've tested a backport of the patch to 2.6 (just replace set_proxy by
_set_proxy in the patch) and the resulting version of urllib2 can login to
the proxy (as expected).

Thanks for the patch.

Fixed and Committed revision 76908 in the trunk.

Fixed through reversions r76908, r76909, r76910, r76911

Thanks for the patch, Tatsuhiro Tsujikawa.

meant revisions.

Hi! 2.6 backport is missing an argument in _set_tunnel definition. It
should be:

    def _set_tunnel(self, host, port=None, headers=None):

The patch fixes only when you pass the authentication info in the proxy
handler's URL. Like:

    proxy_handler = urllib2.ProxyHandler({'https':
'http://user:pass@proxy-example.com:3128/'})

But setting the authentication using a ProxyBasicAuthHandler is still
broken:

    proxy_auth_handler = urllib2.ProxyBasicAuthHandler()
    proxy_auth_handler.add_password('realm', 'proxy-example.com:3128',
'user', 'pass')

In the attached file (urllib2_with_proxy_auth_comparison.py) we've wrote
a comparison between what works with HTTP and HTTPS.

The problem is the 407 error never reaches the ProxyBasicAuthHandler
because HTTPConnection._tunnel raises an exception when the http
response status code is not 200.

Thanks for the note, Manuel. Fixed it in revision 77013.

In this ticket, setting the authentication using a ProxyBasicAuthHandler is not yet addressed yet. (this was informed in the last note). Reopening this one to track it.

I have worked up a monkey patch for urllib2/httplib for the issue of setting the authentication using a Proxy(Basic|Digest)AuthHandler.

The basic approach was to create a new httplib exception (ProxyTunnelError) and raise that with the http response attached so that the HTTPSHandler can determine when 407 Proxy authentication required is present, and then reroute the urllib2.OpenerDirector to error handling mode.

Unfortunately, there is a backwards compatibility issue - cases where a socket.error was previously being raised now get an ProxyTunnelError. Not that you could do much useful with the socket.error in the first place, but I suppose you could look for '407' in the text. Ugh.

If you think this might prove useful, let me know and I can rework it into a real patch - just let me know what branch/version to base it off of. (My monkey patch is for python 2.6.4.)

I've been in contact w/ Barry Scott offline re: the monkey patch previously mentioned. I'm attaching a 2.7 maintenance branch patch that he has needed to extend, and plans to follow up on.

Attached to this comment (can you attach multiple files at once?) is the somewhat moldy 2.6.4 monkey patch, mercilessly ripped from my own code and probably not good for much.

The attached patch builds on Mike's work.

The core of the problem is that the Request object
did not know what was going on. This means that it
was not possible for get_authorization() to work
for proxy-auth and www-auth.

I change Request to know which of the four types of
connection it represents. There are new methods on
Request that return the right information based on
the connection type.

To understand how to make this work I needed to
instrument the code. There is now a set_debuglevel
on the OpenerDirector object that turns on debug in
all the handlers and the director. I have added
more debug messages to help understand this code.

This code now passes the 72 test cases I run. I'll
attach the code I used to test as a follow up to this.

Attached is the code I used to test these changes.
See the README.txt file for details include
the results of a test run.

I left out some white space changes to match the style
of the std lib code. Re posting with white space cleanup.

I've found that for the Python2.6.x patch to play nicely with the popular rquests library, I've had to set some defaults on the modified __init__ function so that it reads as follows:

    def __init__(self, *args, **kwargs):
        _orig_init(self, *args, **kwargs)
        self._tunnel_headers = {}
        self._tunnel_host = ''
        self._tunnel_port = ''

Also seems to work with python 2.6.1. Note: Change the entry condition to:

if os.environ.get('https_proxy', None) and sys.version_info[:2]  == (2, 6) :

Also needed to make another minor update to the monkey patch.
Have uploaded the new files as new_http_proxy_patch.py for use with python 2.6.x

Is there any work still needed here? Surely the 2.6.x patches can't be applied unless there are security issues?

I believe this also affects Python 3; see bpo-24333.

I think making the CONNECT response object available to the caller is the right general approach. But I really dislike raising an exception that holds a socket connection to be closed. (I know this is already done with urllib.error.HTTPError; let’s not repeat this in the “http.client” module!)

Ideally, at the “http.client” level I would prefer to avoid all the special set_tunnel() calls, and use the usual request() and getresponse() API to make the CONNECT request. This way the response status and header fields would be available just like any other response. For this approach to work, we would probably need to add a new HTTPConnection.detach() method that released the original socket reader and writer, and add a way to create a new HTTPConnection instance using these socket objects. This enhancement probably wouldn’t be appropriate for Python 2 or a bug fix release. But it seems the cleanest approach to me, and may also allow using HTTPConnection with the Upgrade header (e.g. for opportunistic encryption, HTTP 2, Web etc), and proxying non-HTTP connections, as bonuses.

A less revolutionary approach might be to add a HTTPSConnection.tunnel() method, that always returns the proxy’s response, but only does the SSL wrapping for a successful response.

For the record, a while ago I think I made a patch implementing my HTTPConnection.detach() proposal. I can probably dig it up if anyone is interested.

However I gave up on fixing this bug in “urllib.request”. As far as I understand it, the framework does not distinguish the 407 Proxy Authentication Required error of the initial proxy CONNECT request from any potential 407 response from a tunnelled connection. Perhaps a special case could be made; I think there are already lots of special cases. But the maze of urlopen() handlers is already too complicated and I decided this was too hard to bother working on. Sorry :)

Modify target versions to bugfix and feature branches

It has been 5 years, now the urllib3 is actively used, but it also inherited this problem: if no authentication data has been received, then the method _tunnel raises an exception OSError that does not contain response headers. Accordingly, this exception cannot be handled. And perhaps this is an obstacle to building a convenient system of authentication on a proxy server in a widely used library requests (it would be nice to be able to just provide an argument proxy_auth, similar to how it is done for server authorization). Now, if a user wants to send a https request through a proxy that requires complex authentication (Kerberos, NTLM, Digest, other) using the urllib3, he must first send a separate request to the proxy, receive a response, extract the necessary data to form the header Proxy-Authorization, then generate this header and pass it to the ProxyManager. And if we are talking about Requests, then the situation there is worse, because you cannot pass proxy headers directly (psf/requests#2708).
If we were to aim to simplify the authentication procedure on the proxy server for the user, then where would we start, do we need to change the http.client so that the error returned by the method _tunnel contains headers? Or maybe it's not worth changing anything at all and the path with preliminary preparation by user of the header Proxy-Authorization is the only correct one? Martin Panter, could you also give your opinion? Thank you in advance.