py2.4.2/win32

The proxy mechanism of python fails on https and does
work completely wrong (not using the CONNECT scheme).

(after urlopen some minute(s) freeze then EOF error)

Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for
more information.
>>> import urllib
>>> urllib.getproxies()
{'ftp': 'ftp://vserver:3128', 'http':
'http://vserver:3128', 'gopher': 'gopher:/
/vserver:3128', 'https': 'https://vserver:3128'}
>>> urllib.urlopen('https://www.elster.de')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Python24\lib\urllib.py", line 77, in urlopen
    return opener.open(url)
  File "C:\Python24\lib\urllib.py", line 185, in open
    return getattr(self, name)(url)
  File "C:\Python24\lib\urllib.py", line 386, in open_https
    h.endheaders()
  File "C:\Python24\lib\httplib.py", line 795, in
endheaders
    self._send_output()
  File "C:\Python24\lib\httplib.py", line 676, in
_send_output
    self.send(msg)
  File "C:\Python24\lib\httplib.py", line 643, in send
    self.connect()
  File "C:\Python24\lib\httplib.py", line 1071, in connect
    ssl = socket.ssl(sock, self.key_file, self.cert_file)
  File "C:\Python24\lib\socket.py", line 74, in ssl
    return _realssl(sock, keyfile, certfile)
IOError: [Errno socket error] (8, 'EOF occurred in
violation of protocol')
>>>



no CONNECT even appears in the squid proxy access log.

Robert
 

Logged In: YES 
user_id=972995

Meanwhile I wrote my own CONNECT quick hack.   As indeed
this hack works correct for all proxied environments tested
up to now (>30) I wonder how open_https  (in urllib and
urllib2) ever in the past managed to come through a proxy,
because there is some differentiation in open_https for the
case, that there is a proxy!? Who has written that
if..else's? Are there proxies which really do
SSL-handshaking directly and make an extra connection to the
target server? I guess that would even make certificate
handling very strange... I cannot immagine and never saw
one. But maybe such proxies exist. I am not a real expert
for such networking questions, but I guess CONNECT is widely
used and in my own proxies I can see in the log file, that
all common browsers use a HTTP CONNECT request for https
proxying. CONNECT should at least be implemented as an
option in urllibX

Robert

seen with urllib2 as well:

https://bugs.launchpad.net/ubuntu/+source/python2.5/+bug/122551

Please verify if this recipe is of any help:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195

The recipe in its current form doesn't work with urllib2 in python 2.5

The reason it fails is that the HTTPConnection.request() method isn't
passed the request itself (with the proxy host and port info). No easy
solution to be seen.

I think this is a major issue because urllib2 is widely used, and any
application using it is useless behind a HTTPS proxy. A Prominent
example in the Linux world is yum, the Fedora Linux package management
tool. HTTPS proxies are a extermely common in the corporate world.

The cookbook recipe shows that it is possible to implement a CONNECT
proxy (with quite a few lines of code, actually), but because it's not
part of urllib2 it has been broken by python 2.5.

Hi,

I am working on a patch to implement the https proxy support for
urllib2. It works fine for me, but feel free to change the patch.

Can any one take a look please?


Thanks

In cast it is not obvious. I already attached the purpose patch
"http-tunnel-urllib" in the bug.

Chris, I'm trying your patch out now. My quick reading of it looks ok.

Well I've happily done lots of SOAP transations, admittedly all to the
same webservice, via HTTPS-over-proxy-CONNECT using ChrisL's patch. It
seems to work just fine. Whom do we petition to get this into the
mainline python sources?
- Cameron

Just supporting, that proxy support over https is widely used and
important. I am wondering, why this request is open for more than 2 years. 
Mathias

I've created a python 2.4 compatible distribution of the python 2.5
urllib2 and httplib modules with the patch attached to this issue. This
has proven especially useful in zc.buildout or other setuptools
environments:

  http://pypi.python.org/pypi/httpsproxy_urllib2

Hi,

looks like the attached patch only works if I use
urllib2.ProxyHandler({'https' : proxy}) at the start, but not when I use
request.set_proxy(proxy, 'https') per request. I tested with Python
2.5.1 on Win32.

Hi NL,

Can you please try this patch on top of the original patch?

Thanks

Chris

--- .pc/fix-up/urllib2.py       2008-02-06 01:13:10.000000000 -0800
+++ urllib2.py  2008-06-27 15:45:34.000000000 -0700
@@ -253,7 +253,7 @@ class Request:
         return self.__r_host

     def set_proxy(self, host, type):
-        if self.type == 'https' and not self._tunnel_host:
+        if self.get_type() == 'https' and not self._tunnel_host:
             self._tunnel_host = self.host
         else:
             self.type = type

On Fri, Jun 27, 2008 at 3:22 PM, nfl <report@bugs.python.org> wrote:
>
> nfl <nflpython@irto.hu> added the comment:
>
> Hi,
>
> looks like the attached patch only works if I use
> urllib2.ProxyHandler({'https' : proxy}) at the start, but not when I use
> request.set_proxy(proxy, 'https') per request. I tested with Python
> 2.5.1 on Win32.
>
> ----------
> nosy: +nfl
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1424152>
> _______________________________________
>

> Can you please try this patch on top of the original patch?

It makes an infinite loop:

Traceback (most recent call last):
  File "F:\nfl\proxyhttps\test.py", line 8, in <module>
    response = urllib2.urlopen(req)
  File "C:\Python25\lib\urllib2.py", line 121, in urlopen
    return _opener.open(url, data)
  File "C:\Python25\lib\urllib2.py", line 377, in open
    req = meth(req)
  File "C:\Python25\lib\urllib2.py", line 1037, in do_request_
    scheme, sel = splittype(request.get_selector())
  File "C:\Python25\lib\urllib2.py", line 253, in get_selector
    return self.__r_host
  File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
    return getattr(self, attr)
  File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
    return getattr(self, attr)
  File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
    return getattr(self, attr)
  File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
    return getattr(self, attr)
  File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
    return getattr(self, attr)
...

The test program was:

import urllib2

targeturl = 'https://www.paypal.com/'
proxyhost = 'proxy.xxxxxxxx.hu:3128'

req = urllib2.Request(targeturl)
req.set_proxy(proxyhost, 'https')
response = urllib2.urlopen(req)
print response.info()

I see that you're working on a final solution, this is great!

What we would need also to incorporate this to the trunk is a patch for
the test suite. Senthil, could you handle this?

>> Senthil, could you handle this?

Sure, I shall take this up, Facundo.

Hi,

After a closer look of your testing program. I believe that your testing
program is not correct. It might be the urllib way of doing things. But
that is not the urllib2 way of doing it. You should add a proxy_handler
rather than directly set_proxy on request objects.

The reason is that, you mess req directly. The urllib2 will not handle
correctly if the request needs rebuild and forward to other handlers.
One example is if the request needs redirect, I believe your test program
will not work.

So the right way of doing things, IMHO, is just adding a proxy handler
which contain the proxy you want. Then that proxy handler will just
automatically set_proxy() to every request it goes out. It even works
if the request needs to be rebuilt.

Then my patch should just work in that case. Try to make it work
for your test script will make things unnecessary complicated.
That defeats the benefit of the handler call chain in urllib2.

Does it make sense to you?

Thanks,

Chris

On Mon, Jun 30, 2008 at 1:49 AM, Nagy Ferenc László
<report@bugs.python.org> wrote:
> The test program was:
>
> import urllib2
>
> targeturl = 'https://www.paypal.com/'
> proxyhost = 'proxy.xxxxxxxx.hu:3128'
>
> req = urllib2.Request(targeturl)
> req.set_proxy(proxyhost, 'https')
> response = urllib2.urlopen(req)
> print response.info()

Mercurial will not work for anyone in a large company without this
fix. I appreciate the patch, but hope its released soon. I did try the
patch with Mercurial, but now I'm getting different error. I'm not
sure if its related to the same bug:

abort: error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol

As indicated by other posters, this *IS A* serious issue with urllib2 as
it does not do CONNECT for HTTPS through Proxy and it fails.

chrisl, I verified your patch and it works properly. I made some minor
changes (make a method private and changes w.r.t code in the trunk) and
also added tests and NEWS to support its inclusion in the trunk. 

Facundo, we should try to include this in py26/py3k, I have attached the
patch for both.

There is a extra patch for test_urllib2net.py which tests real-time
HTTPS connectivity taking the proxies from environment variables
(HTTPS_PROXY). However, that has a serious dependency on Issue1251,
which is still in Open state.  When the bug Issue1251 is fixed, we can
include the  issue1424152-py26-test_urllib2net.diff separately.

Test issue1424152-py26-test_urllib2net.diff and
issue1424152-py3k-test_urllib2net.diff patches has a dependency on
Issue1251 for failure scenarios.

Issue1251 deals with ssl module not support non-blocking handshakes. So,
when the HTTPS environment is NOT SET, while HTTPS Proxy is used, this
test will try to a do_handshake()  in ssl module and will return as it
wont get timed-out. 

This test case can be included after Issue1251 is fixed.

Python 2.4.3 also doesn't support ssl and a proxy to be used at the same
time in CentOS 5.1. This file addresses the issues. It has been tested
by using YUM to access a repository through a proxy using ssl client
certificates. YUM itself needed to be patched to allow for client certs
to be used... that change is on its way upstream.

Thanks to all the other guys here for supplying the first patches!

We need make sure this happens for 2.7/3.1.

As its large enough to be a new feature it won't make it into
2.4/2.5/2.6/3.0.

I agree, gregory. I had verified the fix and supplied patch for py2.6
and py3K. If those still apply to the trunk, I think we should go ahead,
otherwise I shall come with the new patch for 2.7/3k.

Everything looks ok, with one detail, the new set_tunnel() function.

Should this function be public? If yes, new documentation should be
added. If not, it should start with "_". 

I think that it should be public, because we're using it from somewhere
else. So, please provide new patchs for documentation (don't bother to
modify actual code patches).

Thank you!

Has there been any progress on this since January? 


It is causing/affecting:
http://www.selenic.com/mercurial/bts/issue967

These patches (well, the 2.6 patch) works for me.  If there's something
I can do to help get this accepted/applied, I'd be happy to do it.
Obviously, I had no part in these patches, but I can take a stab at
updating documentation or something if that is all that is needed.

Hans, please take a look to my comment 79573 in this same bug. If you
could do that, it'd be amazing...

Thank you!

Sure -- I will tackle that this week; I hope I can come up with
something satisfactory.  As I have not provided doc patches before for
Python I may need to spin up a bit on that (and of course on the exact
meaning of the set_tunnel method in this context) -- but I'm happy to help!

Sorry for the delay. I know that patches work and I had tested them as
well. I shall push those patches to trunk/p3k within this week.

> Facundo Batista <facundo@taniquetil.com.ar> added the comment:
> 
> Hans, please take a look to my comment 79573 in this same bug. If you
> could do that, it'd be amazing...

Oops, had missed that one.. Shall update the patch, Facundo.

reassigning to the volunteer to get this in :)

Fixed for Py2x in the revision 72880. Tested it on a Squid Proxy setup
and found it working fine. Should this be backported?

For Py3k, the Request class has undergone changes since the patch was
developed, I am modifying the patch for Py3k to accommodate those changes.

I added a versionadded: 2.7 note to the docs in r72890.

This change is not suitable for back porting as it arguably adds a new 
feature.

With this patch, I continued to get the following error 
SSL23_GET_SERVER_HELLO
Until my coworker finally found a fix posted by Philippe Biondi:
+++ b/mercurial/keepalive.py
@@ -237,6 +237,8 @@
              else:
                  # no (working) free connections were found.  Create a
new one.
                  h = http_class(host)
+                if hasattr(req,"_tunnel_host") and req._tunnel_host:
+                    h.set_tunnel(req._tunnel_host)
                  if DEBUG: DEBUG.info("creating new connection to %s (%d)",
                                       host, id(h))
                  self._cm.add(host, h, 0)

AndrewTrick: I am assuming your last comment is more relevant to
mercurial's use of the set_tunnel, the facility provided by the patch,
that is solving the issue for you. You had earlier pointed out
mercurial's dependency upon this issue too.

The fix as such stands good and may not require any change. Is my
understanding OK?

I should have pointed out that my secondary problem was a mercurial
dependency on the urllib patch. I just wanted Mercurial users to get a
complete fix. I figure they will be looking for a fix in the python bug
report, and need to be told the fix won't work for them.

On Mon, Jun 22, 2009 at 5:11 PM, Senthil <report@bugs.python.org> wrote:

>
> Senthil <orsenthil@gmail.com> added the comment:
>
> AndrewTrick: I am assuming your last comment is more relevant to
> mercurial's use of the set_tunnel, the facility provided by the patch,
> that is solving the issue for you. You had earlier pointed out
> mercurial's dependency upon this issue too.
>
> The fix as such stands good and may not require any change. Is my
> understanding OK?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1424152>
> _______________________________________
>

@gregory.p.smith:
> This change is not suitable for back porting as it arguably adds a new 
feature.

Speaking as a Mercurial user who can't use Mercurial at work through a
proxy firewall... I beg you to consider that fixing this is not really
adding a "new feature" but fixing a broken implementation requirement.
Surely proxy support is not optional for any serious HTTP library.

> Craig McQueen comment:
>
> Speaking as a Mercurial user who can't use Mercurial at work through a
> proxy firewall... I beg you to consider that fixing this is not really

We might have to take this up at python-dev. I shall do that
to get other developers opinion on backporting this.

Senthil, are you still working on the 3.x version of the patch?

> 
> Senthil, are you still working on the 3.x version of the patch?

Sorry for the delay. I got to work on it. Shall start and shall try
to get it in soon.

+1 for mercurial inoperability from behind proxy

Note that this bug is also of concern to Ubuntu.  See for example
https://bugs.launchpad.net/python/+bug/94130.

This is actually fixed in Py2.7 and I am trying to get it Py3.2 before
marking this as fixed.

Based on the comments, should this be backported to Py2.6?

Personally I would like to see that happen, but I think you should
probably poll python-dev since it can be argued either way as to whether
it is a bug fix or a new feature.  FWIW, from the feeder bug reports it
looks to me like the users perceive it as a bug.

+1 to the above. 

(imo, this is a bugfix, not a new feature)

Personally, I also think that this is a bugfix, though I realize that in
order to fix the bug an API addition had to be made.  So +1 from me on
rolling into 2.6 (though my opinion is simply that of someone who's
eager to see it fixed).

Fixed for 3.2 in py3k branch revision 74198. Asked python-dev for
objections on back porting changes 2.6.

Shall back port changes to py2.6 and py3.1

*Rereads latest patch*

I'm happy with this being backported as a bugfix to 3.1 and 2.6 so long
as the set_tunnel method is hidden.  Call it _set_tunnel() in the
backport (and don't document its existence).

Agreed to the suggestions made by Greg.
The fix is now backported.
For Py2.6 in the revision 74203.
For Py3k in the revision 74204.

Closing this issue.

Unfortunately this bug was only fixed in urllib2.py but never in urllib.py. This keeps biting people in the butt to this very day. Example: https://chromiumcodereview.appspot.com/10825107/

The attached patch remedies that and also fixes some more issues:

(1) proxy_bypass handling:
URLopener.open() will route the request to one of the open_*() methods based on the type of the *request* url. However, if a proxy is defined, it instead routes the request based on the type of the *proxy* url. So far so good. But: In open_http(), the code checks if proxy_bypass(realhost) is true and if so it modifies the Host header of the outgoing request. This code only works properly if the request url is by chance of type "http". If the request url type is e.g. "ftp" and the proxy url type is "http" and there's a proxy_bypass defined for realhost, things will go awry since the program will try to speak HTTP with realhost while it should really speak FTP. (In other words, open_ftp() should be used instead of open_http(), the program is stuck in the wrong codepath.) Also, proxy_bypass handling is currently only implemented for the proxy url type "http" (and not, for instance, "https"). The patch solves this by moving the proxy_bypass check to URLopener.open(): If a proxy_bypass is defined for realhost, the request is routed based on the request url type and not based on the proxy url type.

(2) addinfourl construction:
Upon successful retrieval of the URL, open_http() and open_https() will construct an addinfourl object and return that to the caller. The object is constructed with hard coded url type "http" / "https". So if for instance the request url type is "ftp" and the proxy url type is "http", the addinfourl object will contain a url whose type will have magically changed from "ftp" to "http".

(3) Superfluous code:
At the beginning of open_http() and open_https(), the program sets "user_passwd = None". Directly below is an if-else statement. At the beginning of the else block the program again sets "user_passwd = None".

The patch also works with Python 2.6 save for set_tunnel() in httplib.py, which was called _set_tunnel() in 2.6.

*ping*

Anybody, please consider applying the patch I've submitted August 8th so that this issue gets fixed in Python 2.7's urllib.py.

Thanks so much.

I have a slight fear that this patch could be considered as a feature addition in 2.7 urllib.py, I would like to quell that and ensure that behaviour expectation is consistent when using urllib or urllib2 and latest urllib/request.py modules.

Also, tests + docs can help a lot in speedier reviews.

In the meantime, Issue 24599 has been opened about URLopener(), which I understand is related to the Python 2 “urllib” half of this bug. Since this issue has been closed, perhaps it is best to continue discussion of issue1424152-py27-urllib.diff there instead.