classification
Title: test_bdb fails on non-UTF-8 locale
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, Michael.Felt, miss-islington, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2019-01-18 11:57 by Michael.Felt, last changed 2020-07-06 06:54 by Michael.Felt. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 21136 merged serhiy.storchaka, 2020-06-25 09:51
PR 21144 merged miss-islington, 2020-06-25 11:21
PR 21145 merged miss-islington, 2020-06-25 11:21
Messages (11)
msg333957 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-18 11:57
I see in the bot history that test_bdb is now failing on AIX

https://buildbot.python.org/all/#/builders/161/builds/718/steps/4/logs/stdio

== CPython 3.8.0a0 (heads/master:a37f52436f, Jan 15 2019, 22:53:01) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /home/buildbot/buildarea/3.x.aixtools-aix-power6/build/build/test_python_5177546
== CPU count: 8
== encodings: locale=ISO8859-15, FS=iso8859-15

FYI: it is not failing on the GCC based bot (mine is based on XLC).
msg334633 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-31 17:57
Update:
buildbot@x064:[/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue]./python -m test -v test_bdb
== CPython 3.8.0a0 (heads/master-dirty:0785889468, Jan 30 2019, 16:16:31) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/build/test_python_11862246
== CPU count: 8
== encodings: locale=ISO8859-15, FS=iso8859-15
Run tests sequentially
0:00:00 [1/1] test_bdb
test_down (test.test_bdb.StateTestCase) ... ok
...
test_step_at_return_with_no_trace_in_caller (test.test_bdb.IssuesTestCase) ... ok

======================================================================
FAIL: test_skip (test.test_bdb.StateTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/test_bdb.py", line 731, in test_skip
    tracer.runcall(tfunc_import)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/test_bdb.py", line 448, in __exit__
    self.test_case.fail(err_msg)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/test_bdb.py", line 582, in fail
    raise self.failureException(msg) from None
AssertionError: encoding with 'iso8859-15' codec failed (BdbNotExpectedError: Wrong event type at expect_set item 2, got 'call'
  Expected: ('line', 3, 'tfunc_import')
  Got:      ('call', 11, 'encode'),         ('quit',),)

----------------------------------------------------------------------

Ran 31 tests in 0.392s

Is this somehow related to "encoding" as in AIX is not using utf-8 as default?
msg334711 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-02-01 19:20
OK. New info.

Back in the time of issue 34347 I installed some extra filesets to support UTF-8 on the virtual machine (aka partition) that I have the bot on.

On systems where this fileset is not installed this test does not fail.

Shall dig further - in a bout a week - but I expect it is something very different than whatever bdb is testing.

fyi - the additional fileset (not normally installed afaik) is/are: 
  bos.loc.com.utf           7.1.4.30    C     F    Common Locale Support - UTF-8
  bos.loc.utf.EN_US         7.1.4.30    C     F    Base System Locale UTF Code
msg335722 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-02-16 21:07
OK. I have narrowed it down to this:
when
LANG=en_US.8859-15

the test fails.

root@x064:[/home/root]grep LANG /etc/environment
LANG=en_US

And now it passes.
msg335765 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-02-17 09:12
Some record keeping - to keep track of when this regression first appeared on the bot:

buildername	POWER6 AIX 3.x	Builder
buildnumber	718	Build
codebase		Build
event	push	Change
github_distinct	true	Change
got_revision	a37f52436f9aa4b9292878b72f3ff1480e2606c3	Git
owners	["Christian Heimes <christian@python.org>"]	Build
project	python/cpython	Build
repository	https://github.com/python/cpython	Build
revision	a37f52436f9aa4b9292878b72f3ff1480e2606c3	Build
scheduler	3.x	Scheduler

And the last build without this error:
buildnumber	717	Build
codebase		Build
event	push	Change
github_distinct	true	Change
got_revision	c9f872b0bdce5888f1879fa74e098bf4a05430c5	Git
owners	["Victor Stinner <vstinner@redhat.com>"]	Build
project	python/cpython	Build
repository	https://github.com/python/cpython	Build
revision	c9f872b0bdce5888f1879fa74e098bf4a05430c5	Build
scheduler	3.x	Scheduler

I am hoping, with this info - someone who understands the commit process better than I can help me see the difference between these two builds.

I have tried

git diff c9f872b0bdce5888f1879fa74e098bf4a05430c5  a37f52436f9aa4b9292878b72f3ff1480e2606c3

And this only seems to be a change in something related to ssl (bpo-35746) - so I am not really understanding what (else) has happened.

The behavior seems to be language handling related, and not really bdb or ssl which is where the test and git diff points at.

Ideas and/or suggestions welcome.

(p.s., no longer showing as a bot error as I have changed my LANG setting)
msg335931 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-02-19 11:46
Again - after switching the env variable LANG to the 'default' everything works as expected.

Leaving it open as a regression - because everything was working with a non-default setting.

When I have more time I'll do a git bisect to try and establish the change where it went wrong.
msg372333 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-25 09:49
test_bdb fails on non-UTF-8 locale because Python executes a Python code from the corresponding "encodings" submodule. There are shortcuts for the UTF-8 codec which avoid using the Python code.
msg372346 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-25 11:21
New changeset 94eee69e9b3a7e7d33142a47ffea560beb8f1596 by Serhiy Storchaka in branch 'master':
bpo-35773: Fix test_bdb on non-UTF-8 locales. (GH-21136)
https://github.com/python/cpython/commit/94eee69e9b3a7e7d33142a47ffea560beb8f1596
msg372350 - (view) Author: miss-islington (miss-islington) Date: 2020-06-25 11:37
New changeset 84f9c23a12eb091ac5327ebcba0d63794085b7f8 by Miss Islington (bot) in branch '3.8':
bpo-35773: Fix test_bdb on non-UTF-8 locales. (GH-21136)
https://github.com/python/cpython/commit/84f9c23a12eb091ac5327ebcba0d63794085b7f8
msg372351 - (view) Author: miss-islington (miss-islington) Date: 2020-06-25 11:41
New changeset fdf6872f3893647e52f123ba62431aaf9f24ad3a by Miss Islington (bot) in branch '3.9':
bpo-35773: Fix test_bdb on non-UTF-8 locales. (GH-21136)
https://github.com/python/cpython/commit/fdf6872f3893647e52f123ba62431aaf9f24ad3a
msg373071 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-06 06:54
Thanks for getting back to this. ++1 :)

On 25/06/2020 11:49, Serhiy Storchaka wrote:
> Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment:
>
> test_bdb fails on non-UTF-8 locale because Python executes a Python code from the corresponding "encodings" submodule. There are shortcuts for the UTF-8 codec which avoid using the Python code.
>
> ----------
> nosy: +serhiy.storchaka
> title: test_bdb fails on AIX bot (regression) -> test_bdb fails on non-UTF-8 locale
> versions: +Python 3.10, Python 3.9
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35773>
> _______________________________________
>
History
Date User Action Args
2020-07-06 06:54:26Michael.Feltsetmessages: + msg373071
2020-06-25 12:33:08serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-06-25 11:41:50miss-islingtonsetmessages: + msg372351
2020-06-25 11:37:42miss-islingtonsetmessages: + msg372350
2020-06-25 11:21:46miss-islingtonsetpull_requests: + pull_request20304
2020-06-25 11:21:39miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request20303
2020-06-25 11:21:29serhiy.storchakasetmessages: + msg372346
2020-06-25 09:52:21serhiy.storchakalinkissue41069 dependencies
2020-06-25 09:51:33serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request20296
2020-06-25 09:49:30serhiy.storchakasetnosy: + serhiy.storchaka
title: test_bdb fails on AIX bot (regression) -> test_bdb fails on non-UTF-8 locale
messages: + msg372333

versions: + Python 3.9, Python 3.10
2019-03-16 09:51:33SilentGhostsetnosy: + David.Edelsohn
2019-02-19 11:46:54Michael.Feltsetmessages: + msg335931
2019-02-17 09:12:24Michael.Feltsetmessages: + msg335765
2019-02-16 21:07:10Michael.Feltsetmessages: + msg335722
2019-02-01 19:20:21Michael.Feltsetmessages: + msg334711
2019-01-31 17:57:17Michael.Feltsetmessages: + msg334633
2019-01-18 11:57:58Michael.Feltcreate