New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
urllib/urllib2: HTTPS over (Squid) Proxy fails #42870
Comments
py2.4.2/win32 The proxy mechanism of python fails on https and does (after urlopen some minute(s) freeze then EOF error) Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for
more information.
>>> import urllib
>>> urllib.getproxies()
{'ftp': 'ftp://vserver:3128', 'http':
'http://vserver:3128', 'gopher': 'gopher:/
/vserver:3128', 'https': 'https://vserver:3128'}
>>> urllib.urlopen('https://www.elster.de')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Python24\lib\urllib.py", line 77, in urlopen
return opener.open(url)
File "C:\Python24\lib\urllib.py", line 185, in open
return getattr(self, name)(url)
File "C:\Python24\lib\urllib.py", line 386, in open_https
h.endheaders()
File "C:\Python24\lib\httplib.py", line 795, in
endheaders
self._send_output()
File "C:\Python24\lib\httplib.py", line 676, in
_send_output
self.send(msg)
File "C:\Python24\lib\httplib.py", line 643, in send
self.connect()
File "C:\Python24\lib\httplib.py", line 1071, in connect
ssl = socket.ssl(sock, self.key_file, self.cert_file)
File "C:\Python24\lib\socket.py", line 74, in ssl
return _realssl(sock, keyfile, certfile)
IOError: [Errno socket error] (8, 'EOF occurred in
violation of protocol')
>>> no CONNECT even appears in the squid proxy access log. Robert |
Logged In: YES Meanwhile I wrote my own CONNECT quick hack. As indeed Robert |
seen with urllib2 as well: https://bugs.launchpad.net/ubuntu/+source/python2.5/+bug/122551 |
Please verify if this recipe is of any help: |
The recipe in its current form doesn't work with urllib2 in python 2.5 The reason it fails is that the HTTPConnection.request() method isn't |
I think this is a major issue because urllib2 is widely used, and any The cookbook recipe shows that it is possible to implement a CONNECT |
Hi, I am working on a patch to implement the https proxy support for Can any one take a look please? Thanks |
In cast it is not obvious. I already attached the purpose patch |
Chris, I'm trying your patch out now. My quick reading of it looks ok. |
Well I've happily done lots of SOAP transations, admittedly all to the
|
Just supporting, that proxy support over https is widely used and |
I've created a python 2.4 compatible distribution of the python 2.5 |
Hi, looks like the attached patch only works if I use |
Hi NL, Can you please try this patch on top of the original patch? Thanks Chris --- .pc/fix-up/urllib2.py 2008-02-06 01:13:10.000000000 -0800 def set_proxy(self, host, type):
- if self.type == 'https' and not self._tunnel_host:
+ if self.get_type() == 'https' and not self._tunnel_host:
self._tunnel_host = self.host
else:
self.type = type On Fri, Jun 27, 2008 at 3:22 PM, nfl <report@bugs.python.org> wrote:
|
It makes an infinite loop: Traceback (most recent call last):
File "F:\nfl\proxyhttps\test.py", line 8, in <module>
response = urllib2.urlopen(req)
File "C:\Python25\lib\urllib2.py", line 121, in urlopen
return _opener.open(url, data)
File "C:\Python25\lib\urllib2.py", line 377, in open
req = meth(req)
File "C:\Python25\lib\urllib2.py", line 1037, in do_request_
scheme, sel = splittype(request.get_selector())
File "C:\Python25\lib\urllib2.py", line 253, in get_selector
return self.__r_host
File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
return getattr(self, attr)
File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
return getattr(self, attr)
File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
return getattr(self, attr)
File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
return getattr(self, attr)
File "C:\Python25\lib\urllib2.py", line 215, in __getattr__
return getattr(self, attr)
... The test program was: import urllib2
targeturl = 'https://www.paypal.com/'
proxyhost = 'proxy.xxxxxxxx.hu:3128'
req = urllib2.Request(targeturl)
req.set_proxy(proxyhost, 'https')
response = urllib2.urlopen(req)
print response.info() |
I see that you're working on a final solution, this is great! What we would need also to incorporate this to the trunk is a patch for |
Sure, I shall take this up, Facundo. |
Hi, After a closer look of your testing program. I believe that your testing The reason is that, you mess req directly. The urllib2 will not handle So the right way of doing things, IMHO, is just adding a proxy handler Then my patch should just work in that case. Try to make it work Does it make sense to you? Thanks, Chris On Mon, Jun 30, 2008 at 1:49 AM, Nagy Ferenc László
|
Mercurial will not work for anyone in a large company without this abort: error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown |
As indicated by other posters, this *IS A* serious issue with urllib2 as chrisl, I verified your patch and it works properly. I made some minor Facundo, we should try to include this in py26/py3k, I have attached the There is a extra patch for test_urllib2net.py which tests real-time |
Test bpo-1424152-py26-test_urllib2net.diff and bpo-1251 deals with ssl module not support non-blocking handshakes. So, This test case can be included after bpo-1251 is fixed. |
Python 2.4.3 also doesn't support ssl and a proxy to be used at the same Thanks to all the other guys here for supplying the first patches! |
We need make sure this happens for 2.7/3.1. As its large enough to be a new feature it won't make it into |
I agree, gregory. I had verified the fix and supplied patch for py2.6 |
Everything looks ok, with one detail, the new set_tunnel() function. Should this function be public? If yes, new documentation should be I think that it should be public, because we're using it from somewhere Thank you! |
reassigning to the volunteer to get this in :) |
Fixed for Py2x in the revision 72880. Tested it on a Squid Proxy setup For Py3k, the Request class has undergone changes since the patch was |
I added a versionadded: 2.7 note to the docs in r72890. This change is not suitable for back porting as it arguably adds a new |
With this patch, I continued to get the following error
SSL23_GET_SERVER_HELLO
Until my coworker finally found a fix posted by Philippe Biondi:
+++ b/mercurial/keepalive.py
@@ -237,6 +237,8 @@
else:
# no (working) free connections were found. Create a
new one.
h = http_class(host)
+ if hasattr(req,"_tunnel_host") and req._tunnel_host:
+ h.set_tunnel(req._tunnel_host)
if DEBUG: DEBUG.info("creating new connection to %s (%d)",
host, id(h))
self._cm.add(host, h, 0) |
AndrewTrick: I am assuming your last comment is more relevant to The fix as such stands good and may not require any change. Is my |
I should have pointed out that my secondary problem was a mercurial On Mon, Jun 22, 2009 at 5:11 PM, Senthil <report@bugs.python.org> wrote:
|
@gregory.p.smith:
Speaking as a Mercurial user who can't use Mercurial at work through a |
We might have to take this up at python-dev. I shall do that |
Senthil, are you still working on the 3.x version of the patch? |
Sorry for the delay. I got to work on it. Shall start and shall try |
+1 for mercurial inoperability from behind proxy |
Note that this bug is also of concern to Ubuntu. See for example |
This is actually fixed in Py2.7 and I am trying to get it Py3.2 before Based on the comments, should this be backported to Py2.6? |
Personally I would like to see that happen, but I think you should |
+1 to the above. (imo, this is a bugfix, not a new feature) |
Personally, I also think that this is a bugfix, though I realize that in |
Fixed for 3.2 in py3k branch revision 74198. Asked python-dev for Shall back port changes to py2.6 and py3.1 |
*Rereads latest patch* I'm happy with this being backported as a bugfix to 3.1 and 2.6 so long |
Agreed to the suggestions made by Greg. Closing this issue. |
Unfortunately this bug was only fixed in urllib2.py but never in urllib.py. This keeps biting people in the butt to this very day. Example: https://chromiumcodereview.appspot.com/10825107/ The attached patch remedies that and also fixes some more issues: (1) proxy_bypass handling: (2) addinfourl construction: (3) Superfluous code: The patch also works with Python 2.6 save for set_tunnel() in httplib.py, which was called _set_tunnel() in 2.6. |
*ping* Anybody, please consider applying the patch I've submitted August 8th so that this issue gets fixed in Python 2.7's urllib.py. Thanks so much. |
I have a slight fear that this patch could be considered as a feature addition in 2.7 urllib.py, I would like to quell that and ensure that behaviour expectation is consistent when using urllib or urllib2 and latest urllib/request.py modules. Also, tests + docs can help a lot in speedier reviews. |
In the meantime, bpo-24599 has been opened about URLopener(), which I understand is related to the Python 2 “urllib” half of this bug. Since this issue has been closed, perhaps it is best to continue discussion of bpo-1424152-py27-urllib.diff there instead. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: