This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: urllib2 fails against IIS (urllib2 can't parse 401 reply www-authenticate headers)
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: deronnax, dholth, orsenthil
Priority: normal Keywords:

Created on 2014-09-25 15:48 by deronnax, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (5)
msg227543 - (view) Author: Mathieu Dupuy (deronnax) * Date: 2014-09-25 15:48
When connecting to a IIS server, it replies that:

Unauthorized Server: Microsoft-IIS/7.5
WWW-Authenticate: Digest qop="auth",algorithm=MD5-sess,nonce="+Upgraded+v1fe2ba746797cfd974e85f9f6dbdd6e514ec45becd2d8cf0112c764c676ad4a00f98517bb166e467dcad4b942254bd9b71d447e3529c509d2",charset=utf-8,realm="Digest"
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
Date: Thu, 25 Sep 2014 15:11:03 GMT
Connection: close
Content-Length: 0

which blew python 2.7 utllib2 like this:

File "tut2.py", line 23, in <module>
    response = opener.open('https://exca010.encara.local.ads/ews/Services.wsdl')
  File "/usr/lib64/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 524, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1090, in http_error_401
    host, req, headers)
  File "/usr/lib64/python2.7/urllib2.py", line 973, in http_error_auth_reqed
    return self.retry_http_digest_auth(req, authreq)
  File "/usr/lib64/python2.7/urllib2.py", line 977, in retry_http_digest_auth
    chal = parse_keqv_list(parse_http_list(challenge))
  File "/usr/lib64/python2.7/urllib2.py", line 1259, in parse_keqv_list
    k, v = elt.split('=', 1)
ValueError: need more than 1 value to unpack


urllib2 seems to assume that every www-authenticate header value will be a list of equal-signe-separated tuple.

On python3, the error is different and trigger this http://bugs.python.org/issue2202 (which is soon-to-be-fixed)
msg227569 - (view) Author: Mathieu Dupuy (deronnax) * Date: 2014-09-25 19:52
I filled the bug in a hurry. You have to read
"when connecting to a IIS for a protected resource and replying with 401 for an authentication challenge, it replies this [...]"
msg227586 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2014-09-26 05:51
Yes, urllib2 does not have any support for NTML based authentication.  And it is a long pending feature request too.

For 2.7, the best way to handle this might be, instead of crashing on WWW-Authenticate: Negotiate, which is a valid response from IIS (1). It should detect it and fail with a helpful message to use a 3rdparty handler along with urllib2 [2]

And for 3.5, I think it is worthy to consider adding the support in stdlib.
@Daniel Holth - I see you are the owner of it. If we choose to adopt it, do you give permission to reuse portions of code (with correct attribution) in the stdlib?

1) http://msdn.microsoft.com/en-us/library/ms995330#http-sso-2_topic1
2) https://code.google.com/p/python-ntlm/
msg227692 - (view) Author: Daniel Holth (dholth) * Date: 2014-09-27 16:07
I am not the copyright holder, I only maintain the pypi package for
python-ntlm. I might have 10 lines of my own code in the whole package.

If running on Windows it would be great to have out of the box "native
windows NTLM" which can be done somehow with the win32 module, I
think...

It may also be possible to use Kerberos authentication (NTLMv2) instead
of NTLM in most cases these days; since after Windows 2000 according to
Wikipedia.

On Fri, Sep 26, 2014, at 01:51 AM, Senthil Kumaran wrote:
> 
> Senthil Kumaran added the comment:
> 
> Yes, urllib2 does not have any support for NTML based authentication. 
> And it is a long pending feature request too.
> 
> For 2.7, the best way to handle this might be, instead of crashing on
> WWW-Authenticate: Negotiate, which is a valid response from IIS (1). It
> should detect it and fail with a helpful message to use a 3rdparty
> handler along with urllib2 [2]
> 
> And for 3.5, I think it is worthy to consider adding the support in
> stdlib.
> @Daniel Holth - I see you are the owner of it. If we choose to adopt it,
> do you give permission to reuse portions of code (with correct
> attribution) in the stdlib?
> 
> 1) http://msdn.microsoft.com/en-us/library/ms995330#http-sso-2_topic1
> 2) https://code.google.com/p/python-ntlm/
> 
> ----------
> assignee:  -> orsenthil
> nosy: +dholth
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22496>
> _______________________________________
msg330343 - (view) Author: Mathieu Dupuy (deronnax) * Date: 2018-11-23 17:09
A lot of time has passed and things have changed significantly. We now live in a mostly python3-world (which doesn't have the bug) and Python2 has less than two years before beeing put to sleep.

If nobody opposes, I offer to close this issue that I opened first.
History
Date User Action Args
2022-04-11 14:58:08adminsetgithub: 66686
2018-12-01 13:21:17deronnaxsetstatus: open -> closed
resolution: wont fix
stage: resolved
2018-11-23 17:09:47deronnaxsetmessages: + msg330343
2018-07-11 07:36:25serhiy.storchakasettype: crash -> behavior
2014-09-27 16:07:41dholthsetmessages: + msg227692
2014-09-26 05:51:36orsenthilsetassignee: orsenthil

messages: + msg227586
nosy: + dholth
2014-09-25 19:52:32deronnaxsetmessages: + msg227569
2014-09-25 18:49:27ned.deilysetnosy: + orsenthil
2014-09-25 15:48:51deronnaxcreate