classification
Title: Using non-ascii that require UTF-8 breaks AIX testing
Type: behavior Stage: patch review
Components: IO, Tests Versions: Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Michael.Felt, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2020-07-26 14:07 by Michael.Felt, last changed 2020-07-30 05:36 by Michael.Felt.

Pull Requests
URL Status Linked Edit
PR 21640 merged serhiy.storchaka, 2020-07-27 14:16
Messages (8)
msg374312 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-26 14:07
issue41069 introduces tests for paths/files containing non-ascii characters.

On AIX - since the merge of PR21035 and PR21156 - the bots have been broken, i.e., returning test failed.

commit 700cfa8c90a90016638bac13c4efd03786b2b2a0
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Thu Jun 25 17:56:31 2020 +0300

    bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035)

commit f925407a19eeb9bf5f7640143979638adce2c677
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Thu Jun 25 20:39:12 2020 +0300

    [3.9] bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035). (GH-21156)

    (cherry picked from commit 700cfa8c90a90016638bac13c4efd03786b2b2a0)

++++++++

Sadly, I cannot determine - exactly - where it is going wrong as the verbose results ends (says SUCCESS, but there is an ENV change, so bot says FAILED) as:

----------------------------------------------------------------------

Ran 614 tests in 59.122s

OK (skipped=8)
Warning -- files was modified by test_io
  Before: []
  After:  ['@test_23134518_tmpæ']
test_io failed (env changed) in 59.7 sec

== Tests result: SUCCESS ==

1 test altered the execution environment:
    test_io

Total duration: 59.8 sec
msg374316 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-26 14:32
Thank you for your report (and for all other heroic work for maintaining AIX). Breaking only one test on AIX looks like a good news.

Do you able to run tests manually? Could you please run the following command?

    ./python -m test --list-cases -uall test_io | xargs -n 1 ./python -m test -vuall test_io -m

It may help to determine what exactly the test was broken.
msg374343 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-26 20:56
Neat! extra arguments!!

The warnings - extracted:

== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_27984450▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_fspath_support (test.test_io.CIOTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.026s

OK
Warning -- files was modified by test_io
  Before: []
  After:  ['@test_27984450_tmpæ']
test_io failed (env changed)

+++++
== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_28246552▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_fspath_support (test.test_io.PyIOTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.026s

OK
Warning -- files was modified by test_io
  Before: []
  After:  ['@test_28246552_tmpæ']
test_io failed (env changed)

== Tests result: SUCCESS ==

1 test altered the execution environment:
    test_io

++++

== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_33620404▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_reader_writer_close_error_on_close (test.test_io.CBufferedRWPairTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
Warning -- Unraisable exception
Exception ignored in: <_io.BufferedRWPair object at 0x30689750>
Traceback (most recent call last):
  File "/home/aixtools/cpython/cpython-master/Lib/test/support/__init__.py", line 613, in gc_collect
    gc.collect()
ValueError: flush of closed file
Warning -- Unraisable exception
Exception ignored in: <_io.BufferedWriter>
Traceback (most recent call last):
  File "/home/aixtools/cpython/cpython-master/Lib/test/support/__init__.py", line 613, in gc_collect
    gc.collect()
ValueError: flush of closed file
test_io failed (env changed)

== Tests result: SUCCESS ==

1 test altered the execution environment:
    test_io

Hope this helps!
msg374377 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-27 14:14
There are two errors in test_io, both are not related to AIX.

The first one is in test_fspath_support. It is exposed on non-UTF-8 locales. AIX by accident use ISO8859-1 locale by default.

The second one is in test_reader_writer_close_error_on_close and related to garbage collector. Seems it is only exposed when run this test separately.
msg374379 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-27 14:26
Excellent!!

aixtools@gcc119:[/home/aixtools/cpython/cpython-master]git pr 21640
remote: Enumerating objects: 50, done.
remote: Counting objects: 100% (50/50), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 58 (delta 46), reused 48 (delta 46), pack-reused 8
Unpacking objects: 100% (58/58), done.
From https://github.com/python/cpython
 * [new ref]         refs/pull/21640/head -> pr_21640

aixtools@gcc119:[/home/aixtools/cpython/cpython-master]./python -m test
test_io
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_io passed in 1 min

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1 min
Tests result: SUCCESS

On 27/07/2020 16:16, Serhiy Storchaka wrote:
> Change by Serhiy Storchaka <storchaka+cpython@gmail.com>:
>
>
> ----------
> keywords: +patch
> pull_requests: +20781
> stage:  -> patch review
> pull_request: https://github.com/python/cpython/pull/21640
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue41401>
> _______________________________________
>
msg374406 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-27 17:58
New changeset 67987acd5dc9776f55f4e139e2b3d9e7a6434d9f by Serhiy Storchaka in branch 'master':
bpo-41401: Fix test_fspath_support in test_io. (GH-21640)
https://github.com/python/cpython/commit/67987acd5dc9776f55f4e139e2b3d9e7a6434d9f
msg374409 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-27 18:07
Automatic backport does not work due to changes in the test.support module. Victor, do you mind to backport PR 21640 to 3.9 and 3.8?
msg374598 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-30 05:36
The 'master' branch bot is working again, the 3.9 branch is still broken, and the 3.8 branch seems, as yet, unaffected.
History
Date User Action Args
2020-07-30 05:36:36Michael.Feltsetmessages: + msg374598
2020-07-27 18:07:46serhiy.storchakasetnosy: + vstinner
messages: + msg374409
2020-07-27 17:58:46serhiy.storchakasetmessages: + msg374406
2020-07-27 14:26:12Michael.Feltsetmessages: + msg374379
2020-07-27 14:16:24serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request20781
2020-07-27 14:14:16serhiy.storchakasetmessages: + msg374377
2020-07-26 20:56:14Michael.Feltsetmessages: + msg374343
2020-07-26 14:32:02serhiy.storchakasetmessages: + msg374316
2020-07-26 14:15:16xtreaksetnosy: + serhiy.storchaka
2020-07-26 14:07:03Michael.Feltcreate