classification
Title: docs do not say that urllib uses HTTP_PROXY
Type: Stage: resolved
Components: Documentation Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: Neil Muller, docs@python, kirikaza, orsenthil, python-dev, terry.reedy
Priority: normal Keywords: patch

Created on 2010-08-19 05:44 by kirikaza, last changed 2012-01-10 17:39 by orsenthil. This issue is now closed.

Files
File name Uploaded Description Edit
proxy.py kirikaza, 2010-08-21 06:36
urllib_PROXY.diff Neil Muller, 2010-11-20 16:14 Prefere lower case version, with docs review
Messages (11)
msg114319 - (view) Author: Kirikaza (kirikaza) Date: 2010-08-19 05:44
In practice urllib reads HTTP_PROXY firstly and then if HTTP_PROXY is empty urllib reads http_proxy. Documentation (http://docs.python.org/library/urllib.html) says nothing about HTTP_PROXY.

Maybe it affects all the versions of Python.
msg114439 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-20 18:31
Are you talking about top-level code within the urllib module or code within defined functions. If the former, can you quote or point to the place in the file? If the latter, which functions? Just urlopen or others? Does urllib2.urlopen have the same issue? (That replaces urllib.urlopen in 3.x).

Having answered the above, please suggest a specific change in a specific place in the docs.
msg114443 - (view) Author: Kirikaza (kirikaza) Date: 2010-08-20 19:41
> Are you talking about ...
I have read no line of code from urllib module. I just try to use urllib.urlopen() and I see it uses not only http_proxy but also HTTP_PROXY and urlopen() prefers the latter variable.

Let's consider a two-lines sample proxy.py:
import urllib
urllib.urlopen('http://python.org/ftp/').readlines()

Some tests:

$ http_proxy= HTTP_PROXY= proxy.py
['<!DOCTYPE HTML ... \n']

$ http_proxy=lower:80 HTTP_PROXY= proxy.py
IOError: invalid proxy for http: 'lower:80'

$ http_proxy= HTTP_PROXY=UPPER:80 proxy.py
IOError: invalid proxy for http: 'UPPER:80'

$ http_proxy=lower:80 HTTP_PROXY=UPPER:80 proxy.py
IOError: invalid proxy for http: 'UPPER:80'

> a specific change in a specific place
place: 20.5.1, paragraph 7 (the only one where *_proxy are discussed)
change: add some words about HTTP_PROXY and maybe about FTP_PROXY

> does urllib2.urlopen ... (in 3.x)
I don't have Python 3.x so I can't say anything.
msg114480 - (view) Author: Kirikaza (kirikaza) Date: 2010-08-21 06:36
I missed the print statement in that example... So I attached the file with correct code.
msg121668 - (view) Author: Neil Muller (Neil Muller) Date: 2010-11-20 16:14
The problem is that the getproxies_environment function converts all environment variables to lower case before deciding whether to use the name. This means that whichever ends up last in os.environment will be used.

The attached patch does two things. It changes the behaviour to prefer the all lower case version if it exists, and adds a note to the ProxyHandler description noting that the environment variable used is not case sensitive, although the all lower case version will be preferred.

The behaviour change is debatable, but preferring the lower case name is likely to be less surprising in general.
msg150933 - (view) Author: Neil Muller (Neil Muller) Date: 2012-01-09 11:45
This patch is now more than a year old with no comment. What needs to happen to get this bug fixed?
msg150959 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-09 18:29
Senthil, any comment on this?
msg150960 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-01-09 18:38
Sorry, this escaped for so long. First thing is, having different values for HTTP_PROXY and HTTP_proxy and http_proxy is plain wrong at the user side. They all should be same and in that case the possible problem which Kirikaza  and Neil Muller state is extremely unlikely (and if the user faces any issue, it is due to improper HTTP configuration).

The reason that lowering of environ vars is performed is similar to any string match with input from user where we want to ignore the case sensitivity.

Having said that, I think, I shall just add teh info on lowering and verification of proxy environment to the docs. I am -1 on changing any behavior for this issue.
msg151020 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-01-10 17:30
New changeset 3370fa13ed73 by Senthil Kumaran in branch '3.2':
Issue9637 - Explain in getproxies_environment that <scheme>_proxy environ variable case does not matter.
http://hg.python.org/cpython/rev/3370fa13ed73

New changeset eb028b3c62c9 by Senthil Kumaran in branch 'default':
Merge from 3.2 - Issue9637 - Explain in getproxies_environment that <scheme>_proxy environ variable case does not matter.
http://hg.python.org/cpython/rev/eb028b3c62c9
msg151021 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-01-10 17:35
New changeset 4a30eae3b945 by Senthil Kumaran in branch '2.7':
port to 2.7 - Issue9637 - Details that case of Proxy env var does not matter.
http://hg.python.org/cpython/rev/4a30eae3b945
msg151022 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-01-10 17:39
This is fixed. Thanks for your contribution.
History
Date User Action Args
2012-01-10 17:39:26orsenthilsetstatus: open -> closed
resolution: fixed
messages: + msg151022

stage: needs patch -> resolved
2012-01-10 17:35:22python-devsetmessages: + msg151021
2012-01-10 17:30:40python-devsetnosy: + python-dev
messages: + msg151020
2012-01-09 18:38:39orsenthilsetassignee: docs@python -> orsenthil
messages: + msg150960
2012-01-09 18:29:09terry.reedysetnosy: + orsenthil
messages: + msg150959
2012-01-09 11:45:15Neil Mullersetmessages: + msg150933
2010-11-20 16:14:17Neil Mullersetfiles: + urllib_PROXY.diff

nosy: + Neil Muller
messages: + msg121668

keywords: + patch
2010-11-20 15:48:01Neil Mullersetversions: + Python 3.2, - Python 2.7
2010-08-21 06:36:22kirikazasetfiles: + proxy.py

messages: + msg114480
2010-08-20 19:41:14kirikazasetmessages: + msg114443
2010-08-20 18:31:41terry.reedysetversions: + Python 2.7, - Python 2.6
nosy: + terry.reedy

messages: + msg114439

stage: needs patch
2010-08-19 05:44:27kirikazacreate