classification
Title: FreeBSD: Fatal Python error: get_locale_encoding: failed to get the locale encoding: nl_langinfo(CODESET) failed
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords: patch

Created on 2018-08-29 23:37 by vstinner, last changed 2018-09-03 20:17 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9044 merged vstinner, 2018-09-03 14:51
PR 9045 merged vstinner, 2018-09-03 15:10
PR 9046 merged vstinner, 2018-09-03 15:21
Messages (8)
msg324361 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 23:37
It seems like my recent work on Python initialization broke Python on FreeBSD. Many test_c_locale_coerce tests now fail with:

"Fatal Python error: get_locale_encoding: failed to get the locale encoding: nl_langinfo(CODESET) failed"

Example of failed buildbot build:

https://buildbot.python.org/all/#/builders/87/builds/1340

It seems like this buildbot started to fail at this build:

https://buildbot.python.org/all/#/builders/87/builds/1332

This build has been triggered by this change:

commit 177d921c8c03d30daa32994362023f777624b10d
Author: Victor Stinner <vstinner@redhat.com>
Date:   Wed Aug 29 11:25:15 2018 +0200

    bpo-34485, Windows: LC_CTYPE set to user preference (GH-8988)
    
    On Windows, the LC_CTYPE is now set to the user preferred locale at
    startup: _Py_SetLocaleFromEnv(LC_CTYPE) is now called during the
    Python initialization. Previously, the LC_CTYPE locale was "C" at
    startup, but changed when calling setlocale(LC_CTYPE, "") or
    setlocale(LC_ALL, "").
    
    pymain_read_conf() now also calls _Py_SetLocaleFromEnv(LC_CTYPE) to
    behave as _Py_InitializeCore(). Moreover, it doesn't save/restore the
    LC_ALL anymore.
    
    On Windows, standard streams like sys.stdout now always use
    surrogateescape error handler by default (ignore the locale).

test_c_locale_coerce tests invalid locales like LANG="invalid.ascii". It seems like Python supported such configuration before my changes.

My recent work:

* bpo-34523: filesystem encoding/errors
* bpo-34485: stdio encoding/errors
msg324378 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-30 11:56
I cannot reproduce the issue on my FreeBSD 11.1 VM. I cannot reproduce the issue on Koobs's "CURRENT-amd64%" buildbot neither :-(

Example:

CURRENT-amd64% env -i LC_ALL=invalid LC_CTYPE=invalid LANG=invalid PYTHONUTF8=0 PYTHONCOERCECLOCALE=0 ./python -X utf8=0 -c pass      

(no error)
msg324512 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 12:38
New changeset f01b2a1b84ee08df73a78cf1017eecf15e3cb995 by Victor Stinner in branch 'master':
bpo-34544: Fix setlocale() in pymain_read_conf() (GH-9041)
https://github.com/python/cpython/commit/f01b2a1b84ee08df73a78cf1017eecf15e3cb995
msg324516 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 13:32
My fix didn't work :-(

https://buildbot.python.org/all/#/builders/87/builds/1355

...
Fatal Python error: get_locale_encoding: failed to get the locale encoding: nl_langinfo(CODESET) failed
...
msg324522 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 15:05
New changeset 8ea09110d413829f71d979d8c7073008cb87fb03 by Victor Stinner in branch 'master':
_Py_CoerceLegacyLocale() restores LC_CTYPE on fail (GH-9044)
https://github.com/python/cpython/commit/8ea09110d413829f71d979d8c7073008cb87fb03
msg324527 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 15:32
New changeset 73b00becbdd40f6a80cfa00abf1ae650a2b5e454 by Victor Stinner in branch 'master':
bpo-34544: pymain_read_conf() don't change LC_ALL (GH-9045)
https://github.com/python/cpython/commit/73b00becbdd40f6a80cfa00abf1ae650a2b5e454
msg324533 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 20:17
New changeset 84b0129b5e0a0e22aad22ae8db2e3833a228aa57 by Victor Stinner in branch '3.7':
_Py_CoerceLegacyLocale() restores LC_CTYPE on fail (GH-9044) (GH-9046)
https://github.com/python/cpython/commit/84b0129b5e0a0e22aad22ae8db2e3833a228aa57
msg324534 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-03 20:17
test_c_locale_coerce pass again on AMD64 FreeBSD 10.x Shared 3.x, I close the issue.
History
Date User Action Args
2018-09-03 20:17:51vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg324534

stage: patch review -> resolved
2018-09-03 20:17:11vstinnersetmessages: + msg324533
2018-09-03 15:32:34vstinnersetmessages: + msg324527
2018-09-03 15:21:04vstinnersetpull_requests: + pull_request8508
2018-09-03 15:10:17vstinnersetpull_requests: + pull_request8506
2018-09-03 15:05:23vstinnersetmessages: + msg324522
2018-09-03 14:51:28vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request8505
2018-09-03 13:32:58vstinnersetmessages: + msg324516
2018-09-03 12:38:24vstinnersetmessages: + msg324512
2018-08-30 11:56:10vstinnersetmessages: + msg324378
2018-08-29 23:37:46vstinnercreate