New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Encoding issues with the locale encoding #76736
Comments
Python 3.6 doesn't use the right encoding in os.strerror(), time.stftime(), locale.localeconv(), time.tzname, etc. on macOS, FreeBSD and other platforms. See my fix locale encodings in bpo-29240: commit 7ed7aea, and test_all_locales.py attached to bpo-29240. See also the bpo-31900 for locale.localeconv() encoding issue when LC_NUMERIC encoding is different than the LC_CTYPE encoding. |
I'm not sure that locale.bindtextdomain() uses the right encoding neither. I propose the following fix: diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
index 324b694b83..1de17d3620 100644
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -555,7 +555,7 @@ PyIntl_bindtextdomain(PyObject* self,PyObject*args)
PyErr_SetFromErrno(PyExc_OSError);
return NULL;
}
- result = PyUnicode_DecodeLocale(current_dirname, NULL);
+ result = PyUnicode_DecodeFSDefault(current_dirname);
Py_XDECREF(dirname_bytes);
return result;
} |
Another issue: _Py_DecodeUTF8Ex() creates surrogate pairs with 16-bit wchar_t (on Windows), whereas input bytes should be escaped. I'm quite sure that it's a bug. |
Example of bug on FreeBSD 11: haypo@freebsd$ LC_ALL=C ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, "fr_FR.ISO8859-1"); print(ascii(os.strerror(2)))' 'Fichier ou r\udce9pertoire inexistant' Expected result: haypo@freebsd$ LC_ALL=fr_FR.ISO8859-1 ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, ""); print(ascii(os.strerror(2)))' 'Fichier ou r\xe9pertoire inexistant' |
freebsd$ LC_ALL=fr_FR.ISO8859-1 ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, ""); print(ascii(os.strerror(2)))' I ran manually this test on FreeBSD: it pass on Python 3.6, 3.7 and master. I close the issue.
I created bpo-33928 for that one. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: