Title: TypeError (bytes/str) in distutils command "upload"
Type: behavior Stage:
Components: Distutils Versions: Python 3.0
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, hagen, loewis, tarek
Priority: normal Keywords: patch

Created on 2008-11-17 17:40 by hagen, last changed 2008-11-20 23:54 by amaury.forgeotdarc. This issue is now closed.

File name Uploaded Description Edit
distutils_upload.patch amaury.forgeotdarc, 2008-11-19 00:56
distutils_upload_2.patch amaury.forgeotdarc, 2008-11-19 12:12
Messages (11)
msg75975 - (view) Author: Hagen Fürstenau (hagen) Date: 2008-11-17 17:40
"python3.0 upload" (on an otherwise sane setup script) results
in the following:

Traceback (most recent call last):
  File "", line 5, in <module>
    import py3setup
  File "/home/MP/hagenf/src/pyskein/", line 22, in <module>
  File "/home/MP/hagenf/local/lib/python3.0/distutils/", line
149, in setup
  File "/home/MP/hagenf/local/lib/python3.0/distutils/", line
942, in run_commands
  File "/home/MP/hagenf/local/lib/python3.0/distutils/", line
962, in run_command
"/home/MP/hagenf/local/lib/python3.0/distutils/command/", line
55, in run
    self.upload_file(command, pyversion, filename)
"/home/MP/hagenf/local/lib/python3.0/distutils/command/", line
116, in upload_file
    auth = "Basic " + base64.encodestring(self.username + ":" +
  File "/home/MP/hagenf/local/lib/python3.0/", line 338, in
    raise TypeError("expected bytes, not %s" % s.__class__.__name__)
TypeError: expected bytes, not str
msg75985 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-17 22:03
The username:password string should be encoded, but with which encoding?
msg75991 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-11-17 23:14
Encoding of basic auth has a loooong story. I keep forgetting the
details, but one interpretation is that the it needs to be MIME B or Q
encoded first (i.e. through a header encoding), which then needs to be
base64 encoded. For B or Q, you can chose any IANA-registered encoding.

This is not directly specified, but rather an implicit interpretation of
RFC2617, through reference to RFC2616 by using the TEXT production,
which is defined (in 2.2 of 2616) to be either Q or B encoded, unless it
is iso-8859-1 (which apparently can pass unencoded).

Another interpretation is that it should be UTF-8, based on the general
policy of RFC 2277 mandates UTF-8 for all new protocols.

Yet another interpretation is that RFC2617 just doesn't support
non-ASCII user names and passwords.

It is the latter interpretation that we should implement (perhaps with a
comment that this interpretation is debated). However, it does have the
advantage of being a subset of the two other interpretations (which
otherwise contradict each other). For PyPI, I believe it is safe to
assume that both usernames and passwords are restricted to ASCII.
msg76033 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-19 00:56
I indeed tested that the PyPI user registration refuses non-ascii 
characters in both username and password.

But the reported error is only the first of a bytes/str mess.
See attached patch.

- I chose to encode package metadata with utf-8
- Since I did not set credentials in some .pypirc file, the submission 
fails; my first tests on a sample package returned a 401 error (normal) 
but now I only get 200 return codes. I hope my package will not show up 
in PyPI...

This needs more testing with a real package of a real developer.
msg76039 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-11-19 05:43
I think Hagen's strategy is about right: report one problem at a time.

Of course, it would speed up the process if Hagen would provide a patch
that fixes a number of such issues at one. Without such an effort, it is
likely that distutils in 3.0 just won't work. I don't see that as a
serious problem, since there will be a 3.0.1 release, and a 3.0.2
release, and so on.
msg76040 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-11-19 05:55
Your patch seems to contain a number of unrelated changes, such as
introducing an abbreviation for http.client, and using .reqest() instead
of putrequest/putheader/endheaders/send. What is the rationale for these
changes? If they are unrelated to this issue, they should be removed
from the patch.
msg76044 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-19 12:00
To correct the reported problem, 3 lines are indeed enough.

I just wanted to test my changes, so I ran " bdist upload" on my
favourite package, even if I expect it to fail at the end because I
don't have a valid PyPI account.

Here are the problems I encountered:
- io.StringIO is used but "import io" is missing
- http = http.client.HTTPConnection() fails because the local variable
has the same name as the imported module
- the http body must be a bytes string

I agree that the change around http.request() is not needed.
Here is another patch with less changes.
It is also more correct that the previous one, now I receive a http 401
error which means that the request was at least understood by the server.
msg76054 - (view) Author: Hagen Fürstenau (hagen) Date: 2008-11-19 15:56
I just tested Amaury's patch and it seems to work fine.

There's a similar str/bytes issue with the "register" command, but I'll
open another issue for that.
msg76149 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-20 23:18
Martin, do you still have remarks about this patch?
Can we apply it?
msg76151 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-11-20 23:49
The patch is fine, please apply.
msg76153 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-20 23:54
Fixed in r67308.
Date User Action Args
2008-11-20 23:54:08amaury.forgeotdarcsetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg76153
2008-11-20 23:49:13loewissetkeywords: - needs review
resolution: accepted
messages: + msg76151
2008-11-20 23:18:36amaury.forgeotdarcsetmessages: + msg76149
2008-11-19 15:56:19hagensetmessages: + msg76054
2008-11-19 12:12:00amaury.forgeotdarcsetfiles: + distutils_upload_2.patch
2008-11-19 12:11:16amaury.forgeotdarcsetfiles: - distutils_upload_2.patch
2008-11-19 12:00:21amaury.forgeotdarcsetfiles: + distutils_upload_2.patch
messages: + msg76044
2008-11-19 05:55:19loewissetmessages: + msg76040
2008-11-19 05:43:27loewissetmessages: + msg76039
2008-11-19 00:56:59amaury.forgeotdarcsetkeywords: + needs review, patch
files: + distutils_upload.patch
messages: + msg76033
2008-11-17 23:14:01loewissetnosy: + loewis
messages: + msg75991
2008-11-17 22:03:55amaury.forgeotdarcsetnosy: + amaury.forgeotdarc, tarek
messages: + msg75985
2008-11-17 17:40:03hagencreate