classification
Title: Add a new codec: "locale", the current locale encoding
Type: enhancement Stage: patch review
Components: Library (Lib), Unicode Versions: Python 3.3
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, lemburg, loewis, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2011-12-17 06:13 by vstinner, last changed 2012-02-10 22:34 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
locale_encoding-3.patch vstinner, 2012-02-07 23:38 review
Messages (9)
msg149660 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-17 06:13
To factorize the code and to fix encoding issues in the time module, I added functions to decode/encode from/to the locale encoding: PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() and PyUnicode_EncodeLocale() (issue #13560). During tests, I realized that os.strerror() should also use the current locale encoding.

Do you think that the codec should be exposed in Python?

--

The C functions are used by:

 * the locale module to decode result of locale functions
 * Py_Main() to decode the PYTHONWARNING environment variable (PyUnicode_DecodeFSDefault can be used here, but PyUnicode_DecodeFSDefault would just call PyUnicode_DecodeLocale because the Python codec is not loaded yet, a funny bootstrap issue)
 * PyUnicode_EncodeFSDefault() and PyUnicode_DecodeFSDefault[AndSize]() before the locale encoding is known and the Python codec is fully ready
 * os.strerror() and PyErr_SetFromErrno*() to decode the error message
 * time.strftime() to encode the format and decode the result if the wcsftime() function is not available and on Windows. On Windows, wcsftime() is available but avoided to workaround an encoding issue in the timezone (see the issue #10653)
 * time to decode time.tzname

The codec can be useful for developers interacting with C functions depending on the locale. Examples: strerror(), strftime(), ... Use the filesystem encoding would be wrong for such function because the locale encoding can be changed by setlocale() with LC_CTYPE or LC_ALL. Use the filesystem encoding would lead to mojibake.

Even if the most common usecases of C functions depending on the locale are already covered by the Python standard library, developers may want to bind new functions using ctypes (or something else), and I believe that the locale encoding would be useful for these bindings.

--

The problem with a new codec is that it becomes more difficult to choose the right encoding:

 * filesystem encoding: filenames, directory names, hostname, environment variables, command line arguments
 * mbcs (ANSI code page): (basically, it is just an alias of the filesystem encoding)
 * locale: write bindings for new C functions?

I suppose that this issue can be solve by writing documentation explaining the usage of each codec.

--

Attached patch adds the new locale codec.

The major limitation of the current implementation is that the codec only supports the strict and the surrogateescape error handlers. I don't plan to implement other error handlers because I don't think that they would be useful, but it would be possible to implement them.

--

I would be "nice" to fix os.strerror() and time.strftime() in Python 3.2, but I don't want to fix them because it would require to add the locale codec and I don't want to do such change in a stable version. The issue only concerns few people changing their locale encoding at runtime. I hope that everybody uses UTF-8 and never change their locale encoding to something else ;-)
msg149662 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-17 06:30
# On FreeBSD, Solaris and Mac OS X, b'\xff' can be decoded in
# the C locale. The C locale is something like ISO-8859-1, not
# 7-bit ASCII.

On FreeBSD, it *is* the ISO-8859-1 encoding.
msg149671 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-17 15:46
Patch version 2: improve the test. Try also the user locale encoding if the C locale uses ISO-8859-1 (should improve the code coverage on FreeBSD, Mac OS X and Solaris).
msg149678 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-17 17:05
I tested locale_encoding-2.patch on Linux, FreeBSD and Windows: UTF-8 and ISO-8859-1 locales on Linux and FreeBSD, and the cp1252 ANSI code page on Windows.
msg149946 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-21 00:48
I would be possible to implement incremental decoder with mbsrtowcs() and incremental encoder with wcsrtombs(), by serializing mbstate_t to a long integer (TextIOWrapper.tell() does something like that). The problem is that mbsrtowcs() and wcsrtombs() are "recent" (not always available). It may also be dangerous to allow the user to pass an arbitrary mbstate_t (using .setstate()).
msg150114 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-22 21:57
+ encoding = locale.getpreferredencoding()

It should be locale.getpreferredencoding(False).
msg150122 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-23 00:46
I'm not sure I like this idea. I think it would be nice to see it discussed on python-dev.
msg152819 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-02-07 23:38
> + encoding = locale.getpreferredencoding()
> It should be locale.getpreferredencoding(False).

Fixed in patch version 3.
msg153080 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-02-10 22:34
According to the discussion on the python-dev mailing list, such codec would add too much confusion to users and so it is better to not add it.
http://mail.python.org/pipermail/python-dev/2012-February/116272.html

I close the issue as wont fix.
History
Date User Action Args
2012-02-10 22:34:39vstinnersetstatus: open -> closed
resolution: wont fix
messages: + msg153080
2012-02-07 23:38:44vstinnersetfiles: - locale_encoding-2.patch
2012-02-07 23:38:43vstinnersetfiles: - locale_encoding.patch
2012-02-07 23:38:35vstinnersetfiles: + locale_encoding-3.patch

messages: + msg152819
2011-12-23 00:46:58pitrousetnosy: + pitrou
messages: + msg150122
2011-12-22 21:57:23vstinnersetmessages: + msg150114
2011-12-21 00:48:43vstinnersetmessages: + msg149946
2011-12-17 17:05:38vstinnersetmessages: + msg149678
2011-12-17 16:10:19ezio.melottisetnosy: + lemburg, ezio.melotti

type: enhancement
components: + Unicode
stage: patch review
2011-12-17 15:46:33vstinnersetfiles: + locale_encoding-2.patch

messages: + msg149671
2011-12-17 06:30:18vstinnersetmessages: + msg149662
2011-12-17 06:15:44vstinnersetfiles: + locale_encoding.patch
keywords: + patch
2011-12-17 06:13:45vstinnercreate