classification
Title: Urllib2 authentication memory.
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: BreamoreBoy, axitkhurana, bradobro, cvrebert, dstanek, martin.panter, mvolz, orsenthil, python-dev, r.david.murray, terry.reedy
Priority: normal Keywords: patch

Created on 2009-10-17 15:32 by bradobro, last changed 2016-05-24 16:41 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
auth.patch axitkhurana, 2015-04-15 18:00 review
auth10.patch axitkhurana, 2015-04-15 23:54 review
prior_auth_docs.patch r.david.murray, 2015-04-16 18:57 review
auth11.patch axitkhurana, 2015-04-16 18:58 review
Messages (12)
msg94180 - (view) Author: Brad Olson (bradobro) Date: 2009-10-17 15:32
For each request requiring HTTP authentication, urllib2 submits the
request without authentication, receives the server's 401
error/challenge, then re-submits the request with authentication.

This is compliant behavior. The problem comes in that urllib2 repeats
this for every ensuing request to the same namespace.

At times this is just an inefficiency--every request gets sent twice,
often with POST data (which can be sizeable).

Sometimes, especially with large POST bodies, this causes a connection
failure.

(Mercurial suffers greatly from this (and I have suggested workaround to
that team.)

This isn't non-compliant behavior, but RFC2617 (sections 2, 3.3)
suggests that once an HTTP client authenticates, it pre-emptively send
authentication with ensuing requests.

More analysis and fix suggestions at
bitbucket.org/bradobro/liquidhg/wiki/Home.
msg236620 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-02-25 21:27
@Senthil what is your opinion of this?
msg240367 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-09 19:24
See also issue 19494.  I think this would be a new feature, and it may be that it should leverge the feature added in issue 19494.  The difference here is that we are proposing to allow it to happen automatically after the initial 401, whereas in 19494 we send it pre-emptively even on the first call.
msg241136 - (view) Author: Akshit Khurana (axitkhurana) * Date: 2015-04-15 18:00
Adding a patch for Python 3+

Some notes:

* Adding a new password manager to handle this case
* The new handler added in issue 19494 had couple of issues
  * test passes even if we use the old handler in added test
  * uses `request.host` instead of `request.full_url` as key for password

* The new handler did assume realm = None before and still does.

I'm using the same logic for adding keys for adding authenticated urls/realm as for login credentials in basic auth handler.
msg241180 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-15 23:38
Added some review comments.

I think the urllib documentation does not really explain how to *use* these classes, and it should, but that is a separate issue.
msg241182 - (view) Author: Akshit Khurana (axitkhurana) * Date: 2015-04-15 23:54
Updated patch with review changes
msg241258 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-16 18:56
Here is the doc patch for the design that Akshit and I agreed to after re-consideration of the API.  This eliminates the HTTPBasicPriorAuthHandler in favor of the HTTPPasswdMgrWithPriorAuth password manager and putting support for prior auth on the AbstractBasicAuthHandler based on whether or not the password_mgr supports the is_authenticated and updated_authenticated methods.  This redesign means that the Proxy handler automatically gets support for prior auth.
msg241259 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-16 18:57
Actually attaching the patch this time.
msg241260 - (view) Author: Akshit Khurana (axitkhurana) * Date: 2015-04-16 18:58
Updated code as per docs
msg241271 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-16 20:36
New changeset 1b9e81cb83bc by R David Murray in branch 'default':
#7159: generalize urllib prior auth support.
https://hg.python.org/cpython/rev/1b9e81cb83bc
msg266231 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-05-24 05:56
Should this be closed?  A substantial patch was pushed year ago and I see no indication of further issue.
msg266246 - (view) Author: Brad Olson (bradobro) Date: 2016-05-24 13:02
Yes, Go ahead. Thanks.

On Tue, May 24, 2016 at 1:56 AM, Terry J. Reedy <report@bugs.python.org>
wrote:

>
> Terry J. Reedy added the comment:
>
> Should this be closed?  A substantial patch was pushed year ago and I see
> no indication of further issue.
>
> ----------
> nosy: +terry.reedy
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue7159>
> _______________________________________
>
History
Date User Action Args
2016-05-24 16:41:29terry.reedysetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2016-05-24 13:02:29bradobrosetmessages: + msg266246
2016-05-24 05:56:11terry.reedysetnosy: + terry.reedy
messages: + msg266231
2015-04-16 20:36:34python-devsetnosy: + python-dev
messages: + msg241271
2015-04-16 18:58:39axitkhuranasetfiles: + auth11.patch

messages: + msg241260
2015-04-16 18:57:37r.david.murraysetfiles: + prior_auth_docs.patch

messages: + msg241259
2015-04-16 18:56:39r.david.murraysetmessages: + msg241258
2015-04-15 23:54:55axitkhuranasetfiles: + auth10.patch

messages: + msg241182
2015-04-15 23:38:59r.david.murraysetmessages: + msg241180
2015-04-15 18:00:52axitkhuranasetfiles: + auth.patch
keywords: + patch
messages: + msg241136
2015-04-13 00:34:43axitkhuranasetnosy: + axitkhurana
2015-04-09 19:24:12r.david.murraysetnosy: + r.david.murray
messages: + msg240367

type: resource usage -> enhancement
stage: needs patch
2015-02-25 21:27:33BreamoreBoysetversions: + Python 3.4, Python 3.5, - Python 3.1, Python 3.2
nosy: + BreamoreBoy

messages: + msg236620

type: behavior -> resource usage
2014-03-19 01:22:52martin.pantersetnosy: + martin.panter
2014-03-09 23:29:30mvolzsetnosy: + mvolz
2010-08-18 00:15:33dstaneksetnosy: + dstanek
2010-07-11 10:23:39BreamoreBoysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2009-11-11 08:13:08cvrebertsetnosy: + cvrebert
2009-10-18 02:05:21orsenthilsetassignee: orsenthil

nosy: + orsenthil
2009-10-17 15:32:38bradobrocreate