Title: The new packaging module should not use the locale encoding
Type: Stage:
Components: Library (Lib) Versions: Python 3.3
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, eric.araujo, pitrou, python-dev, tarek, vstinner
Priority: normal Keywords: patch

Created on 2011-05-19 11:54 by vstinner, last changed 2013-01-03 02:57 by vstinner. This issue is now closed.

File name Uploaded Description Edit
packaging_utf8.patch vstinner, 2011-05-19 11:54 review
Messages (10)
msg136280 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-19 11:54
The locale encoding is not portable, packaging should use UTF-8 instead.

Attached patch is a draft to workaround "LANG=C ./python -m test test_packaging" failures. I'm not sure that my test in Metadata.write_file() of packaging.metadata is a good idea.

Moreover, packaging should also maybe use the surrogateescape error handler, but I propose to decide that later and in another issue. We may use this error handler only to read files (like the Python makefile).

For write a complete patch, *all* calls to open() in Lib/packaging/* (including Lib/packaging/tests/*) should be checked.

distutils uses the locale encoding, which is UTF-8 on Mac OS X and most Linux setup (but never on Windows). But distutils cannot be fixed, whereas packaging is a new module and can be fixed.
msg136281 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-19 11:56
See also issue #9561 (distutils: set encoding to utf-8 for input and output files) and #6011 (python doesn't build if prefix contains non-ascii characters).

If you are curious, read also #8611 (Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)) and #9425 (Rewrite import machinery to work with unicode paths).
msg136282 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-19 12:01
Looks good, please commit this
msg136290 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-19 13:19
I ported a distutils fix (for non-ASCII path) into packaging:

New changeset cc5cfeaa4a8d by Victor Stinner in branch 'default':
Issue #10419, issue #6011: port 6ad356525381 fix from distutils to packaging
msg136291 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-19 13:24
+        if not isinstance(fileobject, StringIO):
+            encoding = codecs.lookup(fileobject.encoding).name

IMO you should try to get the "encoding" attribute and silence the AttributeError instead.

(also, I'm not even sure why you're adding this check)
msg136312 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-05-19 16:50
New changeset fe740c1cee02 by Victor Stinner in branch 'default':
Issue #12112: packaging reads and writes setup.cfg using UTF-8

New changeset 01d61096140a by Victor Stinner in branch 'default':
Issue #12112: packaging reads/writes metadata using UTF-8
msg136313 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-19 16:52
What is the MainProgram.inspect_file() function in packaging.create? Which kind of file should it process? => What is the encoding of the input files?
msg136315 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-19 17:04
That's not used anymore, I am going to strip it
msg136330 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-05-19 19:45
New changeset a636cb1b7f84 by Victor Stinner in branch 'default':
Issue #12112: fix the encoding of in the packaging module
msg141384 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-29 14:47
I can run LANG=C python -m test test_packaging successfully.  Can we close this?
Date User Action Args
2013-01-03 02:57:56vstinnersetstatus: open -> closed
resolution: fixed
2011-07-29 14:47:49eric.araujosetmessages: + msg141384
2011-05-19 19:45:25python-devsetmessages: + msg136330
2011-05-19 17:04:54tareksetmessages: + msg136315
2011-05-19 16:52:57vstinnersetmessages: + msg136313
2011-05-19 16:50:26python-devsetnosy: + python-dev
messages: + msg136312
2011-05-19 14:22:08Arfreversetnosy: + Arfrever
2011-05-19 13:24:01pitrousetnosy: + pitrou
messages: + msg136291
2011-05-19 13:19:25vstinnersetmessages: + msg136290
2011-05-19 12:01:48tareksetmessages: + msg136282
2011-05-19 11:56:49vstinnersetmessages: + msg136281
2011-05-19 11:54:46vstinnersetnosy: + tarek, eric.araujo
components: + Library (Lib)
2011-05-19 11:54:27vstinnercreate