classification
Title: urllib2.HTTPDigestAuthHandler fails on third hostname?
Type: behavior Stage: needs patch
Components: Extension Modules Versions: Python 2.6
process
Status: open Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: ajaksu2, cmb, danohuiginn, georg.brandl, jjlee, orsenthil
Priority: normal Keywords: easy, patch

Created on 2008-12-17 14:14 by cmb, last changed 2014-02-03 19:06 by BreamoreBoy.

Files
File name Uploaded Description Edit
urllib2-test.txt cmb, 2008-12-17 14:14 HTTPDigestAuthHandler test script
nonce_count.diff danohuiginn, 2009-09-01 15:38 reset nonce count for new nonces review
Messages (12)
msg77965 - (view) Author: Chris Boyle (cmb) Date: 2008-12-17 14:14
The attached script, when edited to fill in valid LiveJournal
credentials (or InsaneJournal, if you change the URL and user list)
fails for me in a very odd way. It fetches the first two URLs quite
happily, and returns a 401 on the third after querying my password
manager about 6 times (always the third, no matter what the user list
is). Rebuilding the opener each time around the loop doesn't fix it.
Recreating the HTTPDigestAuthHandler (and then rebuilding the opener)
*does* fix it. Seems like any given instance of HTTPDigestAuthHandler
fails on the third hostname.

Python package version 2.5.1-5ubuntu5.2. Haven't tried it on newer versions.
msg78026 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2008-12-18 14:39
Chris,
Is there a chance that this is some sort of protection on LJ's side?
Does a given instance mean the same connection being reused? What
happens with longer sleeps?
msg78027 - (view) Author: Chris Boyle (cmb) Date: 2008-12-18 14:49
When I say a given instance, I mean an HTTPDigestAuthHandler object. If
I take the three lines before my loop (authhandler=, opener=,
install_opener) and move them into the loop, the problem goes away,
whereas moving only the latter two of those doesn't help. So, creating a
new HTTPDigestAuthHandler each time fixes the problem, where surely it
shouldn't make any difference? Also, I've tried sleeps of 30 seconds,
with the same results (reuse handler -> fail on the 3rd request, new
handler each time -> works for all requests).
msg78028 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2008-12-18 15:14
Hmm, notice that AbstractDigestAuthHandler handles retries: 

class AbstractDigestAuthHandler:
    def __init__(self, passwd=None):
        ...
        self.retried = 0
        ...
    def reset_retry_count(self):
        self.retried = 0

    def http_error_auth_reqed(self, auth_header, host, req, headers):
        authreq = headers.get(auth_header, None)
        if self.retried > 5:
            # Don't fail endlessly - if we failed once, we'll probably
            # fail a second time. Hm. Unless the Password Manager is
            # prompting for the information. Crap. This isn't great
            # but it's better than the current 'repeat until recursion
            # depth exceeded' approach <wink>
            raise HTTPError(req.get_full_url(), 401, "digest auth failed",
                            headers, None)
        else:
            self.retried += 1
        if authreq:
            scheme = authreq.split()[0]
            if scheme.lower() == 'digest':
                return self.retry_http_digest_auth(req, authreq)


The fact that your password manager is queried six times suggests you're
seeing the "if self.retried > 5:" above. 


Could you make a local copy of /usr/lib/python2.5/urllib2.py
 and add some prints (specially around get_authorization) to see what is
going on?
msg78043 - (view) Author: Dan (danohuiginn) Date: 2008-12-18 21:55
Reasonable idea, Daniel, but changing self.retried makes no difference.
What does solve Chris' problem (which I can reproduce, btw) is
preventing the nonce_count from incrementing.

i.e. comment out the line :
 self.nonce_count += 1
in AbstractDigestAuthHandler.get_authorization

This makes me wonder if all this is being caused by LJ, intentionally or
not, rejecting repeated requests.
msg78047 - (view) Author: Dan (danohuiginn) Date: 2008-12-18 23:02
Actually, this does look more like an urllib2 bug.
According to RFC 2617, the nonce count should represent the number of
requests sent with a particular nonce. But we don't reset the nonce
count when we start using a new nonce. That discrepancy in nonce counts
causes LJ to reject the connection.

Why does it fail the third time, rather than the second? See the LJ code
from http://code.sixapart.com/svn/livejournal/trunk/cgi-bin/ljlib.pl
(which I *think* applies here, but could easily be wrong):
    # check the nonce count
    # be lenient, allowing for error of magnitude 1
msg92141 - (view) Author: Dan (danohuiginn) Date: 2009-09-01 15:38
Attaching a patch. I don't have a test to go with it, except for the one
submitted with the bug report.
msg95276 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2009-11-15 08:32
Interesting issue. RFC 2617 supports the claim. In RFC 2617, section
3.2.2 The Authorization Request Header, we see that nonce-count is
maintained for each particular nonce value and it can used by the server
to verify the replays.

"""
The nc-value is the hexadecimal count of the number of requests
(including the current request) that the client has sent with the nonce
value in this request.  For  example, in the first request sent in
response to a given nonce value, the client sends "nc=00000001".  The
purpose of this  directive is to allow the server to detect request
replays by  maintaining its own copy of this count - if the same
nc-value is  seen twice, then the request is a replay. 
"""

resetting the nonce_count for each new nonce is the correct thing to do.
Patch seems okay and I going ahead with that.

We don't have test for HTTPDigestAuth yet, the nonce tests should be a
part of that.
msg95278 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2009-11-15 08:51
Fixed in r76288 (trunk), r76289 (release26-maint), r76290 (py3k) and
r76291(release31-maint). The issue is fixed. I am keeping it open to
remind of the pending tests (HTTPAuthDigest and nonce value check testcase).
msg102365 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-04-05 09:38
Ping?
msg102366 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-04-05 09:42
Okay, This vaguely got out of my mind. Shall come with the tests for HTTPAuthDigest.
msg174692 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2012-11-03 21:02
Are the tests for HTTPAuthDigest still outstanding?
History
Date User Action Args
2014-02-03 19:06:47BreamoreBoysetnosy: - BreamoreBoy
2012-11-03 21:02:46BreamoreBoysetnosy: + BreamoreBoy
messages: + msg174692
2010-04-05 09:42:59orsenthilsetmessages: + msg102366
2010-04-05 09:38:45georg.brandlsetnosy: + georg.brandl
messages: + msg102365
2009-11-15 08:51:12orsenthilsetresolution: accepted -> fixed
messages: + msg95278
2009-11-15 08:32:45orsenthilsetresolution: accepted
messages: + msg95276
2009-11-14 16:39:03orsenthilsetassignee: orsenthil
2009-09-01 15:38:02danohuiginnsetfiles: + nonce_count.diff
keywords: + patch
messages: + msg92141
2009-04-22 17:27:48ajaksu2setkeywords: + easy
2009-02-13 01:19:32ajaksu2setnosy: + jjlee, orsenthil
stage: needs patch
versions: + Python 2.6, - Python 2.5
2008-12-18 23:02:35danohuiginnsetmessages: + msg78047
2008-12-18 21:55:58danohuiginnsetnosy: + danohuiginn
messages: + msg78043
2008-12-18 15:15:00ajaksu2setmessages: + msg78028
2008-12-18 14:49:23cmbsetmessages: + msg78027
2008-12-18 14:39:17ajaksu2setnosy: + ajaksu2
messages: + msg78026
2008-12-17 14:14:18cmbcreate