classification
Title: TypeError in urllib when trying to use HTTP authentication
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: Dmitry.Jemerov, ezio.melotti, orsenthil, pitrou
Priority: normal Keywords: patch

Created on 2010-03-12 15:37 by Dmitry.Jemerov, last changed 2010-08-04 17:52 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
8123.patch Dmitry.Jemerov, 2010-07-23 11:15
Messages (9)
msg100938 - (view) Author: Dmitry Jemerov (Dmitry.Jemerov) Date: 2010-03-12 15:37
I'm trying to download a file from a site using HTTP authentication. I'm subclassing FancyURLOpener, returning my credentials from the prompt_user_passwd() method, and using opener.retrieve() to download the file. I get the following error:

  File "C:/JetBrains/IDEA/build/eap/downandup.py", line 36, in download
    opener.retrieve(url, os.path.join(target_path, name))
  File "C:\Python31\lib\urllib\request.py", line 1467, in retrieve
    fp = self.open(url, data)
  File "C:\Python31\lib\urllib\request.py", line 1435, in open
    return getattr(self, name)(url)
  File "C:\Python31\lib\urllib\request.py", line 1609, in open_http
    return self._open_generic_http(http.client.HTTPConnection, url, data)
  File "C:\Python31\lib\urllib\request.py", line 1605, in _open_generic_http
    response.status, response.reason, response.msg, data)
  File "C:\Python31\lib\urllib\request.py", line 1621, in http_error
    result = method(url, fp, errcode, errmsg, headers)
  File "C:\Python31\lib\urllib\request.py", line 1859, in http_error_401
    return getattr(self,name)(url, realm)
  File "C:\Python31\lib\urllib\request.py", line 1931, in retry_http_basic_auth
    return self.open(newurl)
  File "C:\Python31\lib\urllib\request.py", line 1435, in open
    return getattr(self, name)(url)
  File "C:\Python31\lib\urllib\request.py", line 1609, in open_http
    return self._open_generic_http(http.client.HTTPConnection, url, data)
  File "C:\Python31\lib\urllib\request.py", line 1571, in _open_generic_http
    auth = base64.b64encode(user_passwd).strip()
  File "C:\Python31\lib\base64.py", line 56, in b64encode
    raise TypeError("expected bytes, not %s" % s.__class__.__name__)
TypeError: expected bytes, not str

The problem happens because _open_generic_http extracts the user password from the string URL, and passes the string to the b64encode method, which only accepts bytes and not strings. The problem happens with Python 3.1.1 for me, but as far as I can see it's still not fixed in the py3k branch as of now.
msg100966 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-03-12 20:50
Could you provide a script (or even better a unittest) to reproduce the problem?
msg101022 - (view) Author: Dmitry Jemerov (Dmitry.Jemerov) Date: 2010-03-13 20:52
from urllib.request import *

opener = FancyURLopener()
opener.retrieve("http://username:password@google.com/index.html", "index.html")
msg101082 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-03-15 02:20
Yes, that seems to be case. When username:password is present in the url string, opener bytes string not being passed to the base64encode function.
msg111277 - (view) Author: Dmitry Jemerov (Dmitry.Jemerov) Date: 2010-07-23 11:15
Patch (with unittest) attached.
msg112362 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-08-01 17:56
Fixed in r83415 and r83416.
msg112832 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-04 16:25
r83416 broke the 3.1 buildbots.


======================================================================
ERROR: test_userpass_inurl (test.test_urllib.urlopen_HttpTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.1.bolen-ubuntu/build/Lib/test/test_urllib.py", line 197, in test_userpass_inurl
    fp = urlopen("http://user:pass@python.org/")
  File "/srv/buildbot/buildarea/3.1.bolen-ubuntu/build/Lib/test/test_urllib.py", line 33, in urlopen
    return opener.open(url)
  File "/srv/buildbot/buildarea/3.1.bolen-ubuntu/build/Lib/urllib/request.py", line 1468, in open
    return getattr(self, name)(url)
  File "/srv/buildbot/buildarea/3.1.bolen-ubuntu/build/Lib/urllib/request.py", line 1642, in open_http
    return self._open_generic_http(http.client.HTTPConnection, url, data)
  File "/srv/buildbot/buildarea/3.1.bolen-ubuntu/build/Lib/urllib/request.py", line 1612, in _open_generic_http
    headers["Authorization"] =  "Basic %s" % auth
BytesWarning: str() on a bytes instance
msg112843 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-08-04 17:51
Surprising that it was not observed in my system (Ubuntu).

There was mistake where in the bytes value was being coerced to a string.
bauth = "Basic: %s" % auth // with auth was bytes.

Fixed it in r83729 and r83730.
msg112845 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-04 17:52
Yes, you would have had to use "python -bb" to evidence it:

$ ./python -bb
Python 3.2a1+ (py3k:83718M, Aug  4 2010, 17:10:53) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> b"" == ""
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BytesWarning: Comparison between bytes and string
History
Date User Action Args
2010-08-04 17:52:36pitrousetmessages: + msg112845
2010-08-04 17:51:05orsenthilsetstatus: open -> closed

messages: + msg112843
2010-08-04 16:25:48pitrousetstatus: closed -> open
nosy: + pitrou
messages: + msg112832

2010-08-01 17:56:19orsenthilsetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg112362

stage: test needed -> resolved
2010-07-23 11:15:18Dmitry.Jemerovsetfiles: + 8123.patch
keywords: + patch
messages: + msg111277
2010-03-15 02:20:22orsenthilsetnosy: + orsenthil
messages: + msg101082

assignee: orsenthil
resolution: accepted
2010-03-13 20:52:51Dmitry.Jemerovsetmessages: + msg101022
2010-03-12 20:50:57ezio.melottisetpriority: normal

type: behavior
versions: + Python 3.2
nosy: + ezio.melotti

messages: + msg100966
stage: test needed
2010-03-12 15:37:44Dmitry.Jemerovcreate