Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check for systemd locale on startup if current locale is set to POSIX #65567

Closed
ncoghlan opened this issue Apr 27, 2014 · 6 comments
Closed

Check for systemd locale on startup if current locale is set to POSIX #65567

ncoghlan opened this issue Apr 27, 2014 · 6 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@ncoghlan
Copy link
Contributor

BPO 21368
Nosy @ncoghlan, @vstinner, @merwok, @vadmium

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-12-12.11:51:23.847>
created_at = <Date 2014-04-27.19:30:25.777>
labels = ['interpreter-core', 'type-feature']
title = 'Check for systemd locale on startup if current locale is set to POSIX'
updated_at = <Date 2017-12-18.14:36:59.077>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2017-12-18.14:36:59.077>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-12-12.11:51:23.847>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2014-04-27.19:30:25.777>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 21368
keywords = []
message_count = 6.0
messages = ['217313', '217328', '217776', '217813', '282985', '308566']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'vstinner', 'eric.araujo', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21368'
versions = ['Python 3.4', 'Python 3.5']
@ncoghlan
Copy link
Contributor Author

bpo-19977 added "surrogateescape" to the fallback settings for the standard streams if Python 3 appears to be running under the POSIX locale (which Python 3 currently reads as setting a default encoding of ASCII, which is almost certainly wrong on any modern Linux system).

If a modern Linux system is using systemd as the process manager, then there will likely be a "/etc/locale.conf" file providing settings like LANG - due to problematic requirements in the POSIX specification, this file (when available) is likely to be a better "source of truth" regarding the system encoding than the environment where the interpreter process is started, at least when the latter is claiming ASCII as the default encoding.

See http://www.freedesktop.org/software/systemd/man/locale.conf.html for more details.

@ncoghlan ncoghlan added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Apr 27, 2014
@vstinner
Copy link
Member

I don't think that Python should read such configuration file. If you consider that something is wrong here, please report the issue to the C library.

@skrah
Copy link
Mannequin

skrah mannequin commented May 2, 2014

Why is the default encoding of POSIX wrong on a modern Linux system?
Today I installed Debian testing, and the first question of the
installer is to choose between "C" and "English" locales for the
install. This with the remark that the chosen locale will be
the default system locale.

@ncoghlan
Copy link
Contributor Author

ncoghlan commented May 3, 2014

That's the problem - even on UTF-8 systems, Linux programs will often be
started in a misconfigured environment: the POSIX locale. Python 2 doesn't
try to interpret the binary data as text, so it doesn't care. Python 3
*does* care, since it automatically converts several pieces of OS provided
data to text using the locale encoding.

systemd tackles that by adding the extra config file to ensure *all* the
environments it creates get the right config, and to provide a more
reliable "source of truth" as to the actual likely encoding of system
interfaces.

However, a fair bit of groundwork is needed to avoid any innate reliance on
the locale encoding before we can reliably override it by default, so this
issue is unlikely to go anywhere before PEP-432 is implemented (and even
then, actually changing the behaviour would be a separate discussion).

@ncoghlan
Copy link
Contributor Author

While this is still a problem I'm interested in solving, I no longer think reading the systemd locale config file would be a good way to address it.

See bpo-28180 for a more recent discussion of some other alternatives.

@ncoghlan ncoghlan added the type-feature A feature request or enhancement label Dec 12, 2016
@vstinner
Copy link
Member

Follow-up: the PEP-538 (bpo-28180) and PEP-540 (bpo-29240) have been accepted and implemented in Python 3.7!

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @ncoghlan