New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PEP 538: silence locale coercion and compatibility warnings by default? #74750
Comments
This is a follow-up to PEP-538 that reflects the uncertainty over whether or not the warning on *successful* implicit locale coercion is a good idea. The argument for this warning is that it alerts redistributors, system integrators, and application developers to the fact that LC_CTYPE may not be what they expect it to be. The argument against it is that in many, and potentially even most, cases where the warning is omitted, there won't be any subsequent problems, and so the warning qualifies as a false alarm (especially for end users of applications that just happen to be written in Python), and the PEP-538 section in the .37 What's New, together with the fact that "LC_CTYPE=C.UTF-8" (or similar) appears in the process environment can be considered sufficient notice of the change for debugging purposes. The initial PEP-538 implementation at #659 includes the warning, this issue reflects the possibility that we may decide to reverse that decision prior to the release of Python 3.7.0. |
My latest proposal on python-dev:
|
Some relevant mailing list threads on the usability problems posed by issuing the warning by default: python-dev (impact on developer experience): https://mail.python.org/pipermail/python-dev/2017-June/148323.html Fedora's python-devel (build environment warnings due to F26 backport of PEP-538): https://lists.fedorahosted.org/archives/list/python-devel@lists.fedoraproject.org/thread/VSEGOF76XMBJOAO4C2MORNK3I2VIPOTU/ |
The warnings caused something around 27 test failures on "AMD64 FreeBSD CURRENT Non-Debug 3.x" buildbot: 27 tests failed again: |
I wrote a PR to remove PEP-538 warnings: Reference (unpatched): haypo@selma$ env -i LC_ALL=C ./python -c "import locale; print(locale.getpreferredencoding())" haypo@selma$ env -i LC_CTYPE=C ./python -c "import locale; print(locale.getpreferredencoding())" haypo@selma$ env -i LC_ALL=C ./python -c "import locale; print(locale.getpreferredencoding())" With my change: haypo@selma$ env -i ./python -c "import locale; print(locale.getpreferredencoding())" haypo@selma$ env -i LC_CTYPE=C ./python -c "import locale; print(locale.getpreferredencoding())" haypo@selma$ env -i LC_ALL=C ./python -c "import locale; print(locale.getpreferredencoding())" |
As Victor notes above, for systems where no suitable coercion target locale is available, even the "unsupported locale" warning is an issue, since it's only full Unicode text handling that's unsupported in such locales - you can still process ASCII-only and non-text data just fine in such environments (and even Unicode data if you're sufficiently careful). As a matter of future-proofing CPython, we also need to account for the fact that some C implementations (potentially including future versions of glibc) may starting using UTF-8 as the default text encoding in the C locale itself (requiring the use of variants like C.ASCII or C.latin-1 to opt-in to the use of legacy text encodings). I still think the warnings are potentially valuable for system integrators and operating system developers trying to ensure that C.UTF-8 is genuinely pervasive as their default locale, so my counter-proposal to Victor's PR is:
That means:
|
Updated issue title to reflect the fact we're now considering just silencing *all* the warnings by default. |
If this warnings are disabled by default, who enable it? I'm OK to remove them all. Additionally, if PEP-540 is accepted, we can use UTF-8 for |
By having the warnings always compiled in, but off by default, "PYTHONCOERCECLOCALE=warn" becomes a debugging tool for integration failures that we (or end users) suspect might be due to the locale coercion behaviour. It's essentially an even more esoteric variant of tools like "PYTHONINSPECT=1", "python -X faulthandler", "python -v", etc. I already learned something myself based on updating the test cases to cover the opt-in warning model: I initially thought that when we set "LC_ALL=C" we'd get *both* warnings, but we don't (at least with glibc). My working theory is that setting 'LC_ALL=C' causes 'setlocale(LC_CTYPE, "C.UTF-8")' to fail, even if there's otherwise a C.UTF-8 locale available. (I haven't conclusively proven that theory, but it's the most likely point for the locale coercion to be implicitly bypassed) |
As I wrote on python-dev, I would prefer no warning and no option to enable |
OK, based on the latest round of custom buildbot results (e.g. http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%20custom/builds/12/steps/test/logs/stdio ), it looks like the main remaining problems are those covered by bpo-30672, where *BSD platforms handle some locales differently from the way Linux handles them (and Mac OS X then inherits those differences), and bpo-30647 (where attempting to use the UTF-8 locale can cause nl_langinfo to fail). So for the latest iteration on *this* change, I'm going to do the following:
|
OK, PEP-538 is effectively disabled on FreeBSD and Mac OS X now, and the locale coercion and compatibility warnings are off by default everywhere. PYTHONCOERCECLOCALE=warn is explicitly documented as a debugging tool for use when folks already suspect that locale coercion or legacy locale incompatibility might be to blame for particular problems that they're seeing. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: