The Python C API has two very useful functions: _Py_char2wchar() and _Py_wchar2char(). They must be used to handle correctly undecodable byte sequences. _Py_char2wchar() and _Py_wchar2char() use the surrogateescape error handler (PEP 383). _Py_char2wchar() forces also the ASCII encoding on FreeBSD and Solaris when the LC_CTYPE locale is C.

Py_Main() expects an array of wide character strings (wchar_t*) for the command line argument, whereas main() gets an array or byte strings (char*). _Py_char2wchar() must be used to be able to call Py_Main().

I propose the following names:

wchar_t* Py_DecodeLocale(const char* arg, size_t *size);
char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos);

See Python/fileutils.c for more information about these functions.


Python 3.3 has already higher level functions (calling _Py_char2_wchar() and _Py_wchar2char()):

PyObject* PyUnicode_DecodeLocale(const char *str, const char *errors);
PyObject* PyUnicode_EncodeLocale(PyObject *unicode, const char *errors);

But these functions cannot be used before Python is initialized.

How often do people need to convert to do platform independent locale encoding before Python is initialized? Encouraging use of platform dependent wchar_t's seems like a bad idea when PyUnicode abstracts away the difference ever since 3.3 released.

You seem to need wchar_t to call Py_Main and Py_SetProgramName.

I think there's an example in the docs which is wrong, because it appears to pass a char* to Py_SetProgramName:
https://docs.python.org/3.4/extending/embedding.html#very-high-level-embedding

> You seem to need wchar_t to call Py_Main and Py_SetProgramName.

Yes, exactly.

New changeset 93a798c7f270 by Victor Stinner in branch 'default':
Issue #18395: Rename ``_Py_char2wchar()`` to :c:func:`Py_DecodeLocale`, rename
http://hg.python.org/cpython/rev/93a798c7f270

New changeset 94d0e842b9ea by Victor Stinner in branch 'default':
Issue #18395, #22108: Update embedded Python examples to decode correctly
http://hg.python.org/cpython/rev/94d0e842b9ea