classification
Title: test_utf8_mode: test_env_var() fails on AMD64 Fedora Rawhide Clang Installed 3.7
Type: Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: cstratak, fweimer, vstinner
Priority: normal Keywords:

Created on 2019-06-04 16:55 by vstinner, last changed 2019-06-13 00:57 by vstinner. This issue is now closed.

Messages (7)
msg344603 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-04 16:55
AMD64 Fedora Rawhide Clang Installed 3.7:
https://buildbot.python.org/all/#/builders/195/builds/104


FAIL: test_env_var (test.test_utf8_mode.UTF8ModeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.7.cstratak-fedora.installed/build/target/lib/python3.7/test/test_utf8_mode.py", line 93, in test_env_var
    self.assertEqual(out, '0')
AssertionError: '1' != '0'
- 1
+ 0


On the buildbot:

vstinner@python-builder-rawhide$ env|grep -E 'LC_|LANG'
LANG=en_US.UTF-8

vstinner@python-builder-rawhide$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
(...)
LC_ALL=


Extract of the test:

        # Cannot test with the POSIX locale, since the POSIX locale enables
        # the UTF-8 mode
        if not self.posix_locale():
            # PYTHONUTF8 should be ignored if -E is used
            out = self.get_output('-E', '-c', code, PYTHONUTF8='1')
            self.assertEqual(out, '0')

The problem seems to be posix_locale() which fails if the C locale has been coerced by PEP 538:

    POSIX_LOCALES = ('C', 'POSIX')

    def posix_locale(self):
        loc = locale.setlocale(locale.LC_CTYPE, None)
        return (loc in POSIX_LOCALES)

This code doesn't work if LC_CTYPE is already coerced.
msg344604 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-04 16:57
It seems like this buildbot has no locale, maybe they have been removed:

vstinner@python-builder-rawhide$ locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
C.utf8
POSIX
msg344605 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-04 17:00
glibc-all-langpacks was already installed, but glibc-langpack-en was not installed. I installed glibc-langpack-en: it worked around the issue.
msg344607 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-04 17:06
Related failure on "AMD64 Fedora Rawhide Clang Installed 2.7":
https://buildbot.python.org/all/#/builders/186/builds/23

ERROR: test_setlocale_unicode (test.test_locale.TestMiscellaneous)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/2.7.cstratak-fedora.installed/build/target/lib/python2.7/test/test_locale.py", line 500, in test_setlocale_unicode
    user_locale = locale.setlocale(locale.LC_CTYPE, '')
  File "/home/buildbot/buildarea/2.7.cstratak-fedora.installed/build/target/lib/python2.7/locale.py", line 581, in setlocale
    return _setlocale(category, locale)
Error: unsupported locale setting
msg344687 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-05 09:30
The locales disappeared from this buildbot because of a recent glibc change and it's a glibc bug (it's being fixed).

https://bugzilla.redhat.com/show_bug.cgi?id=1716710
msg344689 - (view) Author: Florian Weimer (fweimer) Date: 2019-06-05 09:32
Just to be clear: glibc-all-langpacks normally contains the data from glibc-langpack-en, but a packaging bug caused removal of the /usr/lib/locale/locale-archive file during upgrades.  This only applies to upgrades.  New installations are fine.

Details here: https://bugzilla.redhat.com/show_bug.cgi?id=1716710
msg345441 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 00:57
It seems like test_utf8_mode has a bug, but sadly I'm no longer able to reproduce it and so I cannot test my fix:

diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index bdb93457cf..c22b95d54d 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -22,6 +22,16 @@ class UTF8ModeTests(unittest.TestCase):
     }
 
     def posix_locale(self):
+        try:
+            import _testinternalcapi
+        except ImportError:
+            pass
+        else:
+            pre_config = _testinternalcapi.get_configs()["pre_config"]
+            if pre_config["coerce_c_locale"] > 0:
+                # C locale has been coerced
+                return True
+
         loc = locale.setlocale(locale.LC_CTYPE, None)
         return (loc in POSIX_LOCALES)
 

I close the issue. Reopen it if you are able to reproduce the bug.
History
Date User Action Args
2019-06-13 00:57:52vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg345441

stage: resolved
2019-06-05 09:32:12fweimersetnosy: + fweimer
messages: + msg344689
2019-06-05 09:30:35vstinnersetmessages: + msg344687
2019-06-04 17:06:41vstinnersetmessages: + msg344607
2019-06-04 17:00:49vstinnersetmessages: + msg344605
2019-06-04 16:57:16vstinnersetnosy: + cstratak
messages: + msg344604
2019-06-04 16:55:02vstinnercreate