classification
Title: check -r fails with non-ASCII unicode long_description
Type: behavior Stage: resolved
Components: Distutils, Distutils2 Versions: Python 3.2, Python 3.3, Python 2.7, 3rd party
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Cykooz, alexis, eric.araujo, jason.coombs, python-dev, tarek
Priority: normal Keywords: patch

Created on 2011-10-06 13:22 by Cykooz, last changed 2012-01-03 20:01 by jason.coombs. This issue is now closed.

Files
File name Uploaded Description Edit
trace.log Cykooz, 2011-10-06 13:22 Exception tracelog
setup.py Cykooz, 2011-10-06 13:33 For test
check_rst.patch Cykooz, 2011-10-07 05:08 Patch for Issue13114 (Python 2.7)
Messages (12)
msg145007 - (view) Author: Kirill Kuzminykh (Cykooz) Date: 2011-10-06 13:22
Command `register` (and `check -r` too) raises the exception UnicodeDecodeError if the long_description (stored as unicode) contains not ASCII chars.
This is because the Docutils, called from Distutils, accepts only ASCII or Unicode. But Distutils passes to Docutils text as a `str` (ASCII or UTF-8).

PS: sorry for my English
msg145008 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-06 13:24
Thank you for the report.  Can you give us a short setup.py that reproduces the bug?
msg145011 - (view) Author: Kirill Kuzminykh (Cykooz) Date: 2011-10-06 13:33
> Can you give us a short setup.py that reproduces the bug?

Ok.

Command that reproduces the bug:
python setup.py check -r

PS: 
I use:
Ubuntu 11.04
Python 2.7.1
Docutils 0.8.1
msg145012 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-06 13:40
Your file uses setuptools, which is not part of Python, but I can reproduce the same bug with distutils.
msg145055 - (view) Author: Kirill Kuzminykh (Cykooz) Date: 2011-10-07 05:08
I created the patch (in attachment) for Distutils from Python 2.7.

PS:
Python 3.2 does not contain this error, because there the entire text is stored in unicode.
msg145091 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-07 15:44
Thanks for the quick patch!  I feared it would be something more complicated :)  I will add a test and commit this.  (I added 3.2 and 3.3 to the versions because I will add the test there too, to make sure the code behaves right.)  Can you give me a Latin transliteration of your name so that I can credit you?
msg145123 - (view) Author: Kirill Kuzminykh (Cykooz) Date: 2011-10-07 18:45
Latin transliteration of my name is Kirill Kuzminykh.
msg145221 - (view) Author: Roundup Robot (python-dev) Date: 2011-10-09 06:59
New changeset e16792003597 by Éric Araujo in branch '3.2':
Add tests for Unicode handling in distutils’ check and register (#13114)
http://hg.python.org/cpython/rev/e16792003597
msg145224 - (view) Author: Roundup Robot (python-dev) Date: 2011-10-09 07:00
New changeset 8d837bd8148a by Éric Araujo in branch '2.7':
Fix distutils’ check and register Unicode handling (#13114).
http://hg.python.org/cpython/rev/8d837bd8148a
msg145230 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-09 07:27
Fixed, thanks again!
msg145524 - (view) Author: Roundup Robot (python-dev) Date: 2011-10-14 14:57
New changeset d5fb646e7ce1 by Éric Araujo in branch 'default':
Add tests for Unicode handling in packaging’ check and register (#13114)
http://hg.python.org/cpython/rev/d5fb646e7ce1
msg150523 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2012-01-03 20:01
I recently encountered this error, and in the process of troubleshooting, developed this one-line to reproduce the problem. I'm including it here for posterity (and in case anybody wants to test a given version of Python for the error):

python -c "from distutils.core import setup; setup(name='foo.project', version='1.0', url='http://www.example.com', author='Foo', author_email='foo@example.com', long_description = u'\xc0',)" check -r
History
Date User Action Args
2012-01-03 20:01:05jason.coombssetnosy: + jason.coombs
messages: + msg150523
2011-10-14 14:57:12python-devsetmessages: + msg145524
2011-10-09 07:27:07eric.araujosetstatus: open -> closed
resolution: fixed
messages: + msg145230

stage: needs patch -> resolved
2011-10-09 07:00:11python-devsetmessages: + msg145224
2011-10-09 06:59:49python-devsetnosy: + python-dev
messages: + msg145221
2011-10-07 18:45:45Cykoozsetmessages: + msg145123
2011-10-07 15:44:45eric.araujosetmessages: + msg145091
versions: + Python 3.2, Python 3.3
2011-10-07 11:07:29Cykoozsettype: crash -> behavior
2011-10-07 06:20:32Cykoozsetversions: + 3rd party
2011-10-07 05:44:07Cykoozsetcomponents: + Distutils2
2011-10-07 05:08:43Cykoozsetfiles: + check_rst.patch
versions: - 3rd party, Python 3.2, Python 3.3
messages: + msg145055

components: - Distutils2
keywords: + patch
type: behavior -> crash
2011-10-06 13:40:30eric.araujosetassignee: tarek -> eric.araujo
type: crash -> behavior

components: + Distutils2
title: UnicodeDecodeError in command `register` due to using not ASCII chars in long_description -> check -r fails with non-ASCII unicode long_description
nosy: + alexis
versions: + 3rd party, Python 3.2, Python 3.3
messages: + msg145012
stage: needs patch
2011-10-06 13:33:16Cykoozsetfiles: + setup.py

messages: + msg145011
2011-10-06 13:24:32eric.araujosetmessages: + msg145008
2011-10-06 13:22:41Cykoozcreate