classification
Title: Proxy handling very slow
Type: performance Stage: resolved
Components: Windows Versions: Python 2.6
process
Status: closed Resolution: out of date
Dependencies: Superseder: proxy_bypass in urllib handling of <local> macro
View: 1648102
Assigned To: Nosy List: ciprian.trofin, martin.panter, orsenthil, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2010-03-26 07:33 by ciprian.trofin, last changed 2017-03-07 18:56 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
proxy.png ciprian.trofin, 2010-03-26 08:43 Setting up the Windows proxy
Messages (5)
msg101742 - (view) Author: Ciprian Trofin (ciprian.trofin) Date: 2010-03-26 07:33
After I installed Python 2.6.5, I noticed a drop in performance of web connections via proxy.

This script:
---------------------------------------------------------
import time
import urllib2

timeMark = time.time()
opener = urllib2.build_opener()
textWeb = opener.open("http://www.google.com/").read()
print time.time() - timeMark
---------------------------------------------------------

takes about 60 seconds to complete (consistently)

The same script, run using Python 2.6.2 is completed in less than 1 second  (0.2seconds)

The test system:
- Windows XP SP3
- internet connection via corporate connection, using a proxy (the proxy is set in Control Panel - Internet Options etc)
msg101743 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-03-26 07:55
How are setting up and using your proxy? It would also help you discuss your approach at python-help@python.org and if you see any specific issues, raise it in the bug report.
msg101745 - (view) Author: Ciprian Trofin (ciprian.trofin) Date: 2010-03-26 08:43
I attached a screenshot.

The proxy is on the corporate network. I'm not setting it up, I'm only using it. Internet Explorer needs that setting in order to connect. Firefox needs this proxy set as "manual proxy configuration".

I looked deeper into the situatioan, usign the following (modified) script:

---------------------------------------------------
import time
import urllib2

timeMark = time.time()
opener = urllib2.build_opener()

proxy = urllib2.ProxyHandler({"http" : "http://10.249.1.63:80"})
opener.add_handler(proxy)

textWeb = opener.open("http://www.google.com/").read()

print time.time() - timeMark

---------------------------------------------------

The only difference from the script above: I set up the proxy manually.
I tested the new script in 2.6.5

With Windows proxy active: same problem, long waiting time.
With Windows proxy disabled: fast execution.
msg190074 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2013-05-26 10:57
I can't even try to reproduce this as I've no corporate network as a test bed.  Is this still an issue with Python 2.7 or the reworked urllib in Python 3.x?
msg270673 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-18 01:38
If this is still a problem, you should narrow down what is causing the slowdown. If you interrupt it, what is the stack trace?

My best guess is perhaps there is a bypass hostname setting and a slow or failing DNS lookup. A call to urllib.proxy_bypass() was added in 2.6.4 for Issue 6894, and in 2.6.5 the Windows registry implementation called gethostname() and gethostbyname(). Perhaps this is fixed by Issue 1648102.
History
Date User Action Args
2017-03-07 18:56:45serhiy.storchakasetstatus: pending -> closed
resolution: out of date
stage: resolved
2016-07-18 01:38:59martin.pantersetstatus: open -> pending

superseder: proxy_bypass in urllib handling of <local> macro
components: + Windows, - None
title: Proxy handling -> Proxy handling very slow
nosy: + martin.panter, paul.moore, tim.golden, steve.dower, zach.ware

messages: + msg270673
2014-02-03 17:12:40BreamoreBoysetnosy: - BreamoreBoy
2013-05-26 10:57:16BreamoreBoysetnosy: + BreamoreBoy
messages: + msg190074
2010-03-26 08:43:54ciprian.trofinsetfiles: + proxy.png

messages: + msg101745
2010-03-26 07:55:11orsenthilsetnosy: + orsenthil
messages: + msg101743
2010-03-26 07:33:14ciprian.trofincreate