This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_c_locale_coercion fails on AIX
Type: behavior Stage: resolved
Components: Build, Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: ncoghlan, vstinner
Priority: normal Keywords: easy

Created on 2017-07-03 13:57 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2713 merged ncoghlan, 2017-07-15 11:58
Messages (8)
msg297587 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 13:57
One example:

http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/957/steps/test/logs/stdio

0:08:55 [ 91/406/2] test_c_locale_coercion failed -- running: test_importlib (41 sec)
skipped 'No C-with-UTF-8 locale available'
test_LC_ALL_set_to_C (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_set_to_zero (test.test_c_locale_coercion.LocaleCoercionTests) ... test_test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) ... 
======================================================================
FAIL: test_LC_ALL_set_to_C (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='', PYTHONCOERCECLOCALE=None)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_c_locale_coercion.py", line 309, in _check_c_locale_coercion
    coercion_expected)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_c_locale_coercion.py", line 202, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'iso8859-1', 'stdin_info': 'iso8859-1:surr[140 chars] 'c'} != {'fsencoding': 'ascii', 'stdin_info': 'ascii:surrogateesc[124 chars] 'c'}
- {'fsencoding': 'iso8859-1',
?                  ^^^^^^^^

+ {'fsencoding': 'ascii',
?                 +++ ^

   'lang': '',
   'lc_all': 'c',
   'lc_ctype': '',
-  'stderr_info': 'iso8859-1:backslashreplace',
?                   ^^^^^^^^

+  'stderr_info': 'ascii:backslashreplace',
?                  +++ ^

-  'stdin_info': 'iso8859-1:surrogateescape',
?                  ^^^^^^^^

+  'stdin_info': 'ascii:surrogateescape',
?                 +++ ^

-  'stdout_info': 'iso8859-1:surrogateescape'}
?                   ^^^^^^^^

+  'stdout_info': 'ascii:surrogateescape'}
?                  +++ ^
msg297592 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-07-03 14:21
This should be a fairly straightforward test-only fix: updating the start of https://github.com/python/cpython/blob/master/Lib/test/test_c_locale_coercion.py to set C_LOCALE_STREAM_ENCODING to 'iso8859-1' when "sys.platform.startswith('aix')".
msg297593 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-07-03 14:23
I also belatedly checked your list at https://unicodebook.readthedocs.io/operating_systems.html#locale-encoding, and confirmed that AIX is the only anomalous case you'd previously identified (aside from the odd nominally-ASCII-but-not-really behaviour on FreeBSD, Mac OS X, and Solaris).
msg297701 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-07-05 05:40
Removing the "easy" tag, as I realised this isn't as clearcut a change as I first thought.

Specifically, I'm wondering if in addition to changing the expectations for this particular test, we should *also* be disabling locale coercion on AIX in general.

After all, latin-1 already round trips just as well as UTF-8+surrogateescape does, and IBM don't seem to have any plans in progress to change the default environment on AIX away from latin-1.
msg297702 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-07-05 05:48
Note that as far as I'm aware, AIX doesn't actually provide any of our coercion target locales yet, so the question in my previous comment is about how we want 3.7.x releases to behave on AIX if IBM *do* introduce a C.UTF-8 locale.

The main argument I see in favour of leaving locale coercion enabled is that the latin-1 default will cope with UTF-8 encoded data from Python 3.7+ applications (even if it displays them incorrectly), but may raise UnicodeEncodeError if an application attempts to create arbitrary Unicode filenames, or write arbitrary unicode data to the standard streams.
msg298395 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-07-15 12:51
New changeset f0b6a261bb8cbede42296e0cc70956fb3b9a6cf7 by Nick Coghlan in branch 'master':
bpo-30836: fix test_c_locale_coercion on AIX (GH-2713)
https://github.com/python/cpython/commit/f0b6a261bb8cbede42296e0cc70956fb3b9a6cf7
msg298396 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-07-15 12:52
This particular test passed on the custom AIX buildbot prior to merging (other tests still have issues on AIX, though, so that buildbot remains red overall).
msg298407 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-15 22:26
Thanks for the fix :-)
History
Date User Action Args
2022-04-11 14:58:48adminsetgithub: 75019
2017-07-15 22:26:44vstinnersetmessages: + msg298407
2017-07-15 12:52:43ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg298396

stage: needs patch -> resolved
2017-07-15 12:51:08ncoghlansetmessages: + msg298395
2017-07-15 11:58:12ncoghlansetpull_requests: + pull_request2777
2017-07-09 07:19:41ncoghlansetassignee: ncoghlan
2017-07-05 05:49:20ncoghlansettype: behavior
stage: needs patch
2017-07-05 05:48:21ncoghlansetmessages: + msg297702
2017-07-05 05:40:16ncoghlansetmessages: + msg297701
components: + Build
2017-07-03 14:23:37ncoghlansetmessages: + msg297593
2017-07-03 14:21:02ncoghlansetkeywords: + easy

messages: + msg297592
2017-07-03 13:57:36vstinnercreate