classification
Title: urllib2 proxy does not work in 2.4.3
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 2.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: georg.brandl, jerrykhan, jjlee, lecaros, mniklas, orsenthil
Priority: normal Keywords:

Created on 2006-07-10 08:29 by mniklas, last changed 2009-11-24 10:05 by orsenthil. This issue is now closed.

Messages (13)
msg29117 - (view) Author: Michal Niklas (mniklas) Date: 2006-07-10 08:29
My python app had to retrieve some web pages and while 
our network environment requires proxy it uses urllib2 
opener (source is in attachment). It worked very well 
on older Python interpreters:
ActivePython 2.4.2 Build 248 (ActiveState Corp.) based 
on
Python 2.4.2 (#67, Oct 30 2005, 16:11:18) [MSC v.1310 
32 bit (Intel)] on win32

It works on linux with 2.3 and 2.4.1:
Python 2.4.1 (#2, May 5 2005, 11:32:06)
[GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2

But it does not work with newest 2.4.3 on Linux:
Python 2.4.3 (#1, Jul 10 2006, 09:57:52)
[GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2

Desired result:
isof-mark:~# python2.3 proxy_bug.py
trying http://www.python.org ...
OK.  We have reply from http://www.python.org.
Size: 13757 [b]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 
Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/
xhtml1-transitional.dtd">
<html lang="en" xml:lang="en" xmlns="http://www.w3.org/
1999/xhtml">
<head>
<me
...
 Yorkshire">design by pollenation</a>
</div>
Copyright Š 1990-2006, <a href="psf">Python Software 
Foundation</a><br />
<a href="about/legal">Legal Statements</a>
</div>
</div>
</div>
</body>
</html>


isof-mark:~# /usr/local/bin/python proxy_bug.py
trying http://www.python.org ...
Traceback (most recent call last):
  File "proxy_bug.py", line 37, in ?
    get_page()
  File "proxy_bug.py", line 27, in get_page
    f = urllib2.urlopen(request)
  File "/usr/local/lib/python2.4/urllib2.py", line 
130, in urlopen
    return _opener.open(url, data)
  File "/usr/local/lib/python2.4/urllib2.py", line 
364, in open
    response = meth(req, response)
  File "/usr/local/lib/python2.4/urllib2.py", line 
471, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python2.4/urllib2.py", line 
402, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python2.4/urllib2.py", line 
337, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.4/urllib2.py", line 
480, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, 
hdrs, fp)
urllib2.HTTPError: HTTP Error 407: Proxy 
Authentication Required

I have raported it on ActiveState bug list (http://
bugs.activestate.com/show_bug.cgi?id=47018) while I 
first spot this bug on their destribution but it seems 
that bug is in others distributions too.

Regards,
Michal Niklas
msg29118 - (view) Author: Michal Niklas (mniklas) Date: 2006-07-10 08:41
Logged In: YES 
user_id=226518

Cannot add attachment via upload so I put it here:
#!/usr/bin/python
# -*- coding: cp1250 -*-

import urllib
import urllib2

def get_page():
	url = 'http://www.python.org'
	print "trying %s ..." % (url)

	# Setup proxy & authentication
	proxy = "poczta.heuthes:8080"
	usr1 = "USER"
	pass1 = "PASSWD"
	proxy_handler = urllib2.ProxyHandler({"http" : "http:/
/" + proxy})
	pass_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
	pass_mgr.add_password(None, "http://" + proxy, usr1, 
pass1)
	pass_mgr.add_password(None, proxy, usr1, pass1)
	auth_handler = urllib2.HTTPBasicAuthHandler(pass_mgr)
	proxy_auth_handler = 
urllib2.ProxyBasicAuthHandler(pass_mgr)

	# Now build a new URL opener and install it
	opener = urllib2.build_opener(proxy_handler, 
proxy_auth_handler, auth_handler, urllib2.HTTPHandler)
	urllib2.install_opener(opener)

	request = urllib2.Request(url)
	f = urllib2.urlopen(request)
	data = f.read()
	print "OK.  We have reply from %s.\nSize: %d [b]" % 
(url, len(data))
	if len(data) < 400:
		print data
	else:
		print data[:200]
		print "..."
		print data[-200:]

get_page()
msg29119 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-07-10 12:49
Logged In: YES 
user_id=849994

Can you please try with 2.5b1? A lot of urllib2 related bugs
have been fixed before this release.
msg29120 - (view) Author: Michal Niklas (mniklas) Date: 2006-07-11 06:27
Logged In: YES 
user_id=226518

Tried it with 2.5 beta 1 and it is not better :(

c:\tools\pyscripts\scripts>c:\python25\python2.5
Python 2.5b1 (r25b1:47027, Jun 20 2006, 09:31:33) [MSC 
v.1310 32 bit (Intel)] on win32


c:\tools\pyscripts\scripts>c:\python25\python2.5 
proxy_bug.py
trying http://www.python.org ...
Traceback (most recent call last):
  File "proxy_bug.py", line 37, in <module>
    get_page()
  File "proxy_bug.py", line 27, in get_page
    f = urllib2.urlopen(request)
  File "c:\python25\lib\urllib2.py", line 121, in urlopen
    return _opener.open(url, data)
  File "c:\python25\lib\urllib2.py", line 380, in open
    response = meth(req, response)
  File "c:\python25\lib\urllib2.py", line 491, in 
http_response
    'http', request, response, code, msg, hdrs)
  File "c:\python25\lib\urllib2.py", line 412, in error
    result = self._call_chain(*args)
  File "c:\python25\lib\urllib2.py", line 353, in 
_call_chain
    result = func(*args)
  File "c:\python25\lib\urllib2.py", line 831, in 
http_error_407
    authority, req, headers)
  File "c:\python25\lib\urllib2.py", line 795, in 
http_error_auth_reqed
    return self.retry_http_basic_auth(host, req, realm)
  File "c:\python25\lib\urllib2.py", line 805, in 
retry_http_basic_auth
    return self.parent.open(req)
  File "c:\python25\lib\urllib2.py", line 380, in open
    response = meth(req, response)
  File "c:\python25\lib\urllib2.py", line 491, in 
http_response
    'http', request, response, code, msg, hdrs)
  File "c:\python25\lib\urllib2.py", line 418, in error
    return self._call_chain(*args)
  File "c:\python25\lib\urllib2.py", line 353, in 
_call_chain
    result = func(*args)
  File "c:\python25\lib\urllib2.py", line 499, in 
http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 407: Proxy Authentication 
Required
msg29121 - (view) Author: Michal Niklas (mniklas) Date: 2006-07-13 10:09
Logged In: YES 
user_id=226518

2.5b2 does not work any better:
Python 2.5b2 (r25b2:50512, Jul 11 2006, 10:16:14) [MSC 
v.1310 32 bit (Intel)] on win32

Result is the same as in 2.5b1 :(
msg29122 - (view) Author: Michal Niklas (mniklas) Date: 2006-07-19 11:12
Logged In: YES 
user_id=226518

I have checked that the last wersion my script works with 
is 2.4.2 and copied that version urllib2.py to 2.4.3 Lib 
directory.  It works again.  The only change in urllib2.py 
is in retry_http_basic_auth(), line 723:
 2.4.2
  user,pw = self.passwd.find_user_password(realm, host)
 2.4.3
  user, pw = self.passwd.find_user_password(realm, 
req.get_full_url())

So "host" is replaced by "req.get_full_url()".

Checked again with 2.5b2 and it works!
Probably I destroyed my test environment and tested
it wrong way :(

So the problem is only with 2.4.3.
Previous versions and 2.5b works well.

Regards,
Michal Niklas
msg29123 - (view) Author: John J Lee (jjlee) Date: 2006-07-20 18:09
Logged In: YES 
user_id=261020

You're sure you didn't copy over the urllib2.py from 2.5b2
also?  That might make the bug appear to go away, when
really it's still there.

The way to be sure is to try it on a different machine.

Thanks for your report.
msg29124 - (view) Author: Michal Niklas (mniklas) Date: 2006-07-21 07:59
Logged In: YES 
user_id=226518

I have just installed new virtual machine with Python 2.5b2 
and my program works.  It seems that only 2.4.3 is broken.

Regards,
Michal
msg29125 - (view) Author: JerryKhan (jerrykhan) Date: 2006-11-28 15:17
Hello, 
In my sens in a general manner there is something wrong in the urllib2 http code:
But this may depends on the environment (I am not an expert in urllib)
Here are my tests : using python 2.4.2 on Windows XP
These simple codes failed with a 407 http error :
Example E1:
import urllib2 as URL
a=URL.urlopen("http://lan_apache_url")
print a.read()

OR example E2:
import urllib2 as URL
r=URL.Request("http://lan_apache_url")
a=URL.urlopen(r)
print a.read()

But succeed with urllib example E3
import urllib
a=urllib.urlopen("http://lan_apache_url")
print a.read()

Notice that different code lines are minimal E1 and E3 are close: 
Notice also that I'm try to access a lan apache server which is not behind a Proxy.
 And I don't want to access to any Proxy (like exclusion string in IExplorer)

But I found also that If I try to access to a protected link with HTTPS ... on the LAN, there is not problem. The issue is really on the HTTP interpreter or during the configuration of the URL opener. 

In the same time, some of my programs are able to access to Internet servers using the current Proxy server without any problem. For that, I use:
import urllib2 as URL
URL.install_opener(URL.build_opener( s.https_handler, s.proxy_auth_handler, s.cookie_handler))

Well, I developed a workaround in my programs ... to use urllib instead of urllib2 in the case where I try to access the LAN (in fact when I don't want to configure the Proxy server, or when the URL match my own proxy exclusion list.)

I espect this will help python urllib2 experts to find the issue.
Jérôme Vacher alias jerrykhan the foolish dracomorpheus of the emerald dragon dynasty.
msg29126 - (view) Author: José Lecaros Cisterna (lecaros) Date: 2007-01-03 16:33
i have the same issue on windows xp, python 2.4.3
but using DOMAIN\username format

msg29127 - (view) Author: John J Lee (jjlee) Date: 2007-01-03 19:37
lecaros and jerrykhan: Do you guys by any chance have a registry key HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyOverride?

I'm guessing this setting is causing urllib to avoid using your default proxy for hosts on your local network, thereby saving you the 407 (the 407 means your proxy is complaining that you've not succeeded in authenticating).

If so, the difference between urllib and urllib2's behaviours does not imply a bug, but just that urllib2 is missing support for getting proxy overrides from the Windows registry.  This could easily be added.
msg29128 - (view) Author: José Lecaros Cisterna (lecaros) Date: 2007-01-03 20:02
I have that key set to <local>, but I don't know what this mean :)  Use for local OR don't use for local.

msg95667 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2009-11-24 10:05
HTTP Proxy Authentication works on Windows and has been verified. Under
the situation of failure, we do not enough details. Closing this bug as
out-of-date as the reporter informed it worked on py2.5 and problem was
with py2.4.
History
Date User Action Args
2009-11-24 10:05:46orsenthilsetstatus: open -> closed
assignee: orsenthil
resolution: out of date
messages: + msg95667
2009-02-12 17:57:18ajaksu2setnosy: + orsenthil
stage: test needed
type: behavior
versions: + Python 2.6, - Python 2.4
2006-07-10 08:29:55mniklascreate