Author vstinner
Recipients Naman-Bhalla, barry, benjamin.peterson, doko, ezio.melotti, jaysinh.shukla, mrabarnett, ncoghlan, paul.moore, serhiy.storchaka, steve.dower, tim.golden, vstinner, xtreak, zach.ware
Date 2019-03-01.01:13:06
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1551402786.77.0.656123222048.issue29571@roundup.psfhosted.org>
In-reply-to
Content
AppVeyor failed on the backport to Python 3.7 of my fix: PR 12108.

Ok, now I understand the bug in Python 3.7. locale.getlocale(locale.LC_CTYPE)[1] returns None because Python doesn't set LC_CTYPE to the user preferred locale. I'm not sure of which locale is used in practice in that case, but at least I can say that None is not the expected encoding name... str.encode() and bytes.decode() use UTF-8 when None is passed as the encoding. locale.getpreferredencoding() returns 'cp1252' which is the ANSI code page.

Python 3.8 is different. In bpo-34485, I modified Python 3.8 to set LC_CTYPE locale to the user preference (ANSI code page):
---
commit 177d921c8c03d30daa32994362023f777624b10d
Author: Victor Stinner <vstinner@redhat.com>
Date:   Wed Aug 29 11:25:15 2018 +0200

    bpo-34485, Windows: LC_CTYPE set to user preference (GH-8988)
    
    On Windows, the LC_CTYPE is now set to the user preferred locale at
    startup: _Py_SetLocaleFromEnv(LC_CTYPE) is now called during the
    Python initialization. Previously, the LC_CTYPE locale was "C" at
    startup, but changed when calling setlocale(LC_CTYPE, "") or
    setlocale(LC_ALL, "").
    
    pymain_read_conf() now also calls _Py_SetLocaleFromEnv(LC_CTYPE) to
    behave as _Py_InitializeCore(). Moreover, it doesn't save/restore the
    LC_ALL anymore.
    
    On Windows, standard streams like sys.stdout now always use
    surrogateescape error handler by default (ignore the locale).
---
History
Date User Action Args
2019-03-01 01:13:06vstinnersetrecipients: + vstinner, barry, doko, paul.moore, ncoghlan, tim.golden, benjamin.peterson, ezio.melotti, mrabarnett, zach.ware, serhiy.storchaka, steve.dower, jaysinh.shukla, Naman-Bhalla, xtreak
2019-03-01 01:13:06vstinnersetmessageid: <1551402786.77.0.656123222048.issue29571@roundup.psfhosted.org>
2019-03-01 01:13:06vstinnerlinkissue29571 messages
2019-03-01 01:13:06vstinnercreate