Calling Py_DecodeLocale() before Py_Initialize() or _Py_InitializeCore() is broken since Python 3.7 if the C locale coercion (PEP-538) or UTF-8 mode (PEP-540) changes the encoding in the middle of _Py_InitializeCore().

I added a new phase to the Python initialization in bpo-36142, a new _PyPreConfig structure, which can be used to fix this mojibake issue.

The code for embedding Python should look like:
---
_Py_PreInitialize();

_PyCoreConfig config;
config.home = Py_DecodeLocale("/path/to/home");

_PyInitError err = _Py_InitializeFromConfig(&config);
if (_Py_INIT_FAILED(err)) {
    _PyCoreConfig_Clear(&config);
    _Py_ExitInitError(err);
}

/* use Python here */

Py_Finalize();
_PyCoreConfig_Clear(&config);

Except that there is no _Py_PreInitialize() function yet :-)

The "vim" editor embeds Python. It sets the Python home by calling Py_SetPythonHome() with the following code:
---

	    size_t len = mbstowcs(NULL, (char *)p_py3home, 0) + 1;

	    /* The string must not change later, make a copy in static memory. */
	    py_home_buf = (wchar_t *)alloc(len * sizeof(wchar_t));
	    if (py_home_buf != NULL && mbstowcs(
			    py_home_buf, (char *)p_py3home, len) != (size_t)-1)
		Py_SetPythonHome(py_home_buf);

ref: https://github.com/vim/vim/blob/14816ad6e58336773443f5ee2e4aa9e384af65d2/src/if_python3.c#L874-L887

mbstowcs() uses the current LC_CTYPE locale. Python can select a different filesystem encoding than the LC_CTYPE encoding depending on PEP-538 and PEP-540. So encoding back the Python home to bytes to access to files on the filesystem can fail because of mojibake.

The code should by written like (pseudo-code):
---
_Py_PreInitialize();

_PyCoreConfig config;
config.home = Py_DecodeLocale(p_py3home);
if (config.home == NULL) { /* ERROR */ }

_PyInitError err = _Py_InitializeFromConfig(&config);
if (_Py_INIT_FAILED(err)) {
    _PyCoreConfig_Clear(&config);
    _Py_ExitInitError(err);
}

The vim case has been discussed at:
https://discuss.python.org/t/adding-char-based-apis-for-unix/916/8

By the way, according to Nick Coghlan (author of the PEP-538), Py_Initialize() and Py_Main() called from an application embedding Python should not coerce the C locale.

They weren't *intended* to change it, and didn't in the original implementation of the PEP, but they do in the as-shipped Python 3.7 implementation, and I abandoned my attempts to revert to the as-designed behaviour as impractical given the other changes made for PEP-540. So that's a behavior we're stuck with now: they both have global side effects on the locale of the calling process.

Victor and I were discussing the appropriate behaviour for the "What do we do if _Py_PreInitialize() hasn't been called?" case, and Victor pointed out that the potential for mojibake provides a solid rationale for going back to the Python 3.6 behaviour: disable both C locale coercion *and* UTF-8 mode, and instead leave locale management to the embedding application.

That would also get us back to the originally intended behaviour of PEP-538, where it only happens by default in the CLI app, not the runtime support library. Back when I decided that doing that would be too complicated, the _Py_PreInitialize API didn't exist yet, and it didn't occur to me that the mojibake problem would affect UTF-8 mode as well.

New changeset d929f18 by Victor Stinner in branch 'master':
bpo-36443: Disable C locale coercion and UTF-8 Mode by default (GH-12589)
d929f18

My commit disabled C locale coercion and UTF-8 Mode by default when Python is embedded which fix this issue in Python 3.8. I close the issue.