Title: urllib URLopener().open https url returns 501 Not Implemented when https_proxy env var is http://
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 2.7
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: l, martin.panter, r.david.murray, stefano-m
Priority: normal Keywords:

Created on 2015-07-09 18:14 by stefano-m, last changed 2020-07-27 16:36 by stefano-m. This issue is now closed.

Messages (10)
msg246515 - (view) Author: Stefano Mazzucco (stefano-m) Date: 2015-07-09 18:14

at work, I am behind a proxy (squid) that is only available over http. So, I have to configure both the http_proxy and https_proxy environment variables to be something like ""

Now, when I try and use urllib to open an "https" url, the proxy returns "501 Not Implemented".

The quick and dirty script below is a minimal demonstration of the problem that appears on both python 2.7 (tested on 2.7.6, 2.7.9, 2.7.10) and 3.4 (tested on 3.4.0 and 3.4.4)

    import urllib
    opener = urllib.URLopener()
except AttributeError:
    # Python 3
    import urllib.request
    opener = urllib.request.URLopener()

url = ''

print("Trying to open", url)

Changing the url to "http://" works OK.


-- Stefano
msg246517 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-07-09 18:35
It is not clear from your description if you actually tested it with python3.  In python2, I believe urllib does not support this (see issue 1424152) while urllib2 does.

Assuming you have, I wonder if the not implemented error is your squid saying it doesn't support CONNECT (which would be a bit surprising, granted).  Have you looked at the error text or the squid logs?
msg246519 - (view) Author: Stefano Mazzucco (stefano-m) Date: 2015-07-09 19:12
I have run the minimal example provided on both Python2 and Python3 with the same results. Sorry if that was not clear.

I did look at issue 1424152 but it seemed to me that I was experiencing a different problem.

When I try and open the page, I get a squid error page with a somewhat vague error saying that "the method is not supported" (even though it's a simple GET and I can get the same page with other tools like wget or a web browser). Unfortunately, I don't have access to the proxied environment right now, and I will need to ask for the squid logs anyway since I can't access them.

I have to say that I have experienced this problem while using buildout as uses urllib.urlretrieve. Surprisingly, it succeeds on Python3, but it fails with Python2 which is our supported version (so there's currently no way that I can use Python3 at work).
msg246535 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-07-10 00:00
David: the original patch made in Issue 1424152 fixed Python 2’s urllib.request.urlopen() and Python 2’s urllib2.urlopen(). But Stefano is using URLopener, which I understand comes from Python 2’s older “urllib” module.

When I run the demonstration, the request to the proxy looks like this:

Accept-Encoding: identity
Connection: close
User-Agent: Python-urllib/3.4

I think Stefano requires a “CONNECT” request instead. There is apparently a patch which sounds like it does this. See issue1424152-py27-urllib.diff and the messages beginning with <>. I suggest any further work (e.g. tests and documentation) continue here, since the other issue has been closed and mainly discusses “urllib2”.
msg246554 - (view) Author: Stefano Mazzucco (stefano-m) Date: 2015-07-10 10:13
Martin, thanks for elaborating my thoughts!

I have dug I bit deeper in Python2's urllib code with pdb, and I think I have narrowed the issue down to what open_http does.

In my example code, replacing with opener.open_http(url) gives the same problem.

I realize I did not provide you with the output of the script, so here it is:

* Python 2.7.10

('Trying to open', '')
Traceback (most recent call last):
  File "", line 30, in <module>
    opener.open_http((host, selector))
  File "/home/mazzucco/.pyenv/versions/2.7.10/lib/python2.7/", line 364, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/home/mazzucco/.pyenv/versions/2.7.10/lib/python2.7/", line 381, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "/home/mazzucco/.pyenv/versions/2.7.10/lib/python2.7/", line 386, in http_error_default
    raise IOError, ('http error', errcode, errmsg, headers)
IOError: ('http error', 501, 'Not Implemented', <httplib.HTTPMessage instance at 0x7f875a67b950>)

* Python 3.4.3

Trying to open
Traceback (most recent call last):
  File "", line 30, in <module>
    opener.open_http((host, selector))
  File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/", line 1805, in open_http
    return self._open_generic_http(http.client.HTTPConnection, url, data)
  File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/", line 1801, in _open_generic_http
    response.status, response.reason, response.msg, data)
  File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/", line 1821, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/", line 1826, in http_error_default
    raise HTTPError(url, errcode, errmsg, headers, None)
urllib.error.HTTPError: HTTP Error 501: Not Implemented

When I unwrap the contents of httplib.HTTPMessage, the error page returned by the squid proxy says:

The requested URL could not be retrieved

The following error was encountered while trying to retrieve the URL:

    Unsupported Request Method and Protocol

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

Looking at Python2's implementation of URLopener's open_http, I can get an even more minimal failing example limited to httplib:

import httplib

host = ''  # this is not the actual proxy

selector = ''

print("Trying to open", selector)

h = httplib.HTTP(host)
h.putrequest('GET', selector)
h.putheader('User-Agent', 'Python-urllib/1.17')
errcode, errmsg, headers = h.getreply()

print(errcode, errmsg)

Running the script on Python 2.7.10 prints:

('Trying to open', '')
(501, 'Not Implemented')
[('content-length', '3069'), ('via', '1.0 (squid/3.1.6)'), ('x-cache', 'MISS from'), ('content-language', 'en'), ('x-squid-error', 'ERR_UNSUP_REQ 0'), ('x-cache-lookup', 'NONE from'), ('vary', 'Accept-Language'), ('server', 'squid/3.1.6'), ('proxy-connection', 'close'), ('date', 'Fri, 10 Jul 2015 09:27:14 GMT'), ('content-type', 'text/html'), ('mime-version', '1.0')]

As I said, I found out about this when using buildout to download files over HTTPS.

Buildout uses urllib.urlretrieve on Python2 and urllib.request.urlretrieve on Python3. I guess that the latter has been fixed in issue 1424152, so that's why I can download with buildout on Python3.
msg246556 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-07-10 11:26
Perhaps you might be able to test out the patch <> to see if that fixes your problem? Though there is a good chance the patch needs updating, since it is fairly old.
msg246557 - (view) Author: Stefano Mazzucco (stefano-m) Date: 2015-07-10 11:58

I have applied the patch <> to my Python2.7.10 installation and seem to work OK.
msg246563 - (view) Author: Lukas Wunner (l) Date: 2015-07-10 13:30
Thank you Martin for referencing my patch. It still applies cleanly with --fuzz=0 to 2.7.10. Would be awesome if this fix would finally get merged.
msg247429 - (view) Author: Stefano Mazzucco (stefano-m) Date: 2015-07-26 14:58
Any thoughts from the core Python developers?

It seems to me that this is a confirmed bug with a working fix (that may need further review, but indeed works). So, hopefully, this issue could be resolved fairly quickly.
msg374401 - (view) Author: Stefano Mazzucco (stefano-m) Date: 2020-07-27 16:36
Closing as this bug refers to versions of Python that have been EOL'd.
Date User Action Args
2020-07-27 16:36:21stefano-msetstatus: open -> closed
resolution: wont fix
messages: + msg374401

stage: resolved
2015-07-26 14:58:16stefano-msetmessages: + msg247429
2015-07-10 13:30:13lsetnosy: + l
messages: + msg246563
2015-07-10 11:58:10stefano-msetmessages: + msg246557
2015-07-10 11:26:46martin.pantersetmessages: + msg246556
2015-07-10 10:13:55stefano-msetmessages: + msg246554
2015-07-10 00:00:37martin.pantersetnosy: + martin.panter
messages: + msg246535
2015-07-09 23:55:46martin.panterlinkissue1424152 dependencies
2015-07-09 19:12:21stefano-msetmessages: + msg246519
2015-07-09 18:35:44r.david.murraysetnosy: + r.david.murray
messages: + msg246517
2015-07-09 18:14:15stefano-mcreate