To decode byte string from the locale encoding (LC_CTYPE), PyUnicode_DecodeFSDefault() can be used, but this function uses a constant encoding set at startup (the locale encoding at startup). The right method is currently to call _Py_char2wchar() and then PyUnicode_FromWideChar(). _Py_char2wchar() is a low level function, it doesn't raise nice Python exception, but just return NULL on error and write a message to stderr using fprintf() (!).

Attached patch adds PyUnicode_DecodeLocale() and PyUnicode_DecodeLocaleAndSize() to offer a high level API to decode data from the *current* locale encoding. These functions fail with an OSError  or MemoryError if decoding fails (instead of a generic ValueError), and don't write to stderr anymore. They are a surrogateescape argument to choose to escape undecodable bytes or to fail with an error.

The patch only uses the function in _localemodule.c, but other functions may have to be fixed to use the new function. The tzname_encoding.patch of issue #5905 should maybe use it for example.

I fixed issue #5905 (strptime fails in non-UTF locale). The fix is not enough if the locale is changed in Python.

Update the patch to fix time.strftime() (if wcsftime() is not available).

changeset:   74002:279b0aee0cfb
user:        Victor Stinner <victor.stinner@haypocalc.com>
date:        Fri Dec 16 23:56:01 2011 +0100
files:       Doc/c-api/unicode.rst Include/unicodeobject.h Modules/_localemodule.c Modules/main.c Modules/timemodule.c
description:
Add PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale()

 * PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale() decode a string
   from the current locale encoding
 * _Py_char2wchar() writes an "error code" in the size argument to indicate
   if the function failed because of memory allocation failure or because of a
   decoding error. The function doesn't write the error message directly to
   stderr.
 * Fix time.strftime() (if wcsftime() is missing): decode strftime() result
   from the current locale encoding, not from the filesystem encoding.

New changeset 88198b93ff2f by Victor Stinner in branch 'default':
Issue #13560: Add PyUnicode_EncodeLocale()
http://hg.python.org/cpython/rev/88198b93ff2f

New changeset 51412b4b81ae by Victor Stinner in branch 'default':
Issue #13560: os.strerror() now uses the current locale encoding instead of UTF-8
http://hg.python.org/cpython/rev/51412b4b81ae

New changeset 07802351ccad by Victor Stinner in branch 'default':
Issue #13560: Locale codec functions use the classic "errors" parameter,
http://hg.python.org/cpython/rev/07802351ccad

Ok, I think that the current code is good enough to close the issue. I opened a more global issue about the Python codec: #13619.