classification
Title: Using non-ascii that require UTF-8 breaks AIX testing
Type: behavior Stage: resolved
Components: IO, Tests Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, Michael.Felt, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2020-07-26 14:07 by Michael.Felt, last changed 2020-12-31 12:45 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 21640 merged serhiy.storchaka, 2020-07-27 14:16
PR 22133 merged vstinner, 2020-09-07 14:16
PR 22135 merged vstinner, 2020-09-07 14:47
PR 24024 closed serhiy.storchaka, 2020-12-31 12:45
Messages (13)
msg374312 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-26 14:07
issue41069 introduces tests for paths/files containing non-ascii characters.

On AIX - since the merge of PR21035 and PR21156 - the bots have been broken, i.e., returning test failed.

commit 700cfa8c90a90016638bac13c4efd03786b2b2a0
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Thu Jun 25 17:56:31 2020 +0300

    bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035)

commit f925407a19eeb9bf5f7640143979638adce2c677
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Thu Jun 25 20:39:12 2020 +0300

    [3.9] bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035). (GH-21156)

    (cherry picked from commit 700cfa8c90a90016638bac13c4efd03786b2b2a0)

++++++++

Sadly, I cannot determine - exactly - where it is going wrong as the verbose results ends (says SUCCESS, but there is an ENV change, so bot says FAILED) as:

----------------------------------------------------------------------

Ran 614 tests in 59.122s

OK (skipped=8)
Warning -- files was modified by test_io
  Before: []
  After:  ['@test_23134518_tmpæ']
test_io failed (env changed) in 59.7 sec

== Tests result: SUCCESS ==

1 test altered the execution environment:
    test_io

Total duration: 59.8 sec
msg374316 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-26 14:32
Thank you for your report (and for all other heroic work for maintaining AIX). Breaking only one test on AIX looks like a good news.

Do you able to run tests manually? Could you please run the following command?

    ./python -m test --list-cases -uall test_io | xargs -n 1 ./python -m test -vuall test_io -m

It may help to determine what exactly the test was broken.
msg374343 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-26 20:56
Neat! extra arguments!!

The warnings - extracted:

== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_27984450▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_fspath_support (test.test_io.CIOTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.026s

OK
Warning -- files was modified by test_io
  Before: []
  After:  ['@test_27984450_tmpæ']
test_io failed (env changed)

+++++
== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_28246552▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_fspath_support (test.test_io.PyIOTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.026s

OK
Warning -- files was modified by test_io
  Before: []
  After:  ['@test_28246552_tmpæ']
test_io failed (env changed)

== Tests result: SUCCESS ==

1 test altered the execution environment:
    test_io

++++

== CPython 3.10.0a0 (heads/master-dirty:b1a8730, Jul 26 2020, 14:00:34) [GCC 7.2.0]
== AIX-2-00F9C1964C00-powerpc-32bit big-endian
== cwd: /home/aixtools/cpython/cpython-master/build/test_python_33620404▒
== CPU count: 32
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_reader_writer_close_error_on_close (test.test_io.CBufferedRWPairTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
Warning -- Unraisable exception
Exception ignored in: <_io.BufferedRWPair object at 0x30689750>
Traceback (most recent call last):
  File "/home/aixtools/cpython/cpython-master/Lib/test/support/__init__.py", line 613, in gc_collect
    gc.collect()
ValueError: flush of closed file
Warning -- Unraisable exception
Exception ignored in: <_io.BufferedWriter>
Traceback (most recent call last):
  File "/home/aixtools/cpython/cpython-master/Lib/test/support/__init__.py", line 613, in gc_collect
    gc.collect()
ValueError: flush of closed file
test_io failed (env changed)

== Tests result: SUCCESS ==

1 test altered the execution environment:
    test_io

Hope this helps!
msg374377 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-27 14:14
There are two errors in test_io, both are not related to AIX.

The first one is in test_fspath_support. It is exposed on non-UTF-8 locales. AIX by accident use ISO8859-1 locale by default.

The second one is in test_reader_writer_close_error_on_close and related to garbage collector. Seems it is only exposed when run this test separately.
msg374379 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-27 14:26
Excellent!!

aixtools@gcc119:[/home/aixtools/cpython/cpython-master]git pr 21640
remote: Enumerating objects: 50, done.
remote: Counting objects: 100% (50/50), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 58 (delta 46), reused 48 (delta 46), pack-reused 8
Unpacking objects: 100% (58/58), done.
From https://github.com/python/cpython
 * [new ref]         refs/pull/21640/head -> pr_21640

aixtools@gcc119:[/home/aixtools/cpython/cpython-master]./python -m test
test_io
0:00:00 Run tests sequentially
0:00:00 [1/1] test_io
test_io passed in 1 min

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1 min
Tests result: SUCCESS

On 27/07/2020 16:16, Serhiy Storchaka wrote:
> Change by Serhiy Storchaka <storchaka+cpython@gmail.com>:
>
>
> ----------
> keywords: +patch
> pull_requests: +20781
> stage:  -> patch review
> pull_request: https://github.com/python/cpython/pull/21640
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue41401>
> _______________________________________
>
msg374406 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-27 17:58
New changeset 67987acd5dc9776f55f4e139e2b3d9e7a6434d9f by Serhiy Storchaka in branch 'master':
bpo-41401: Fix test_fspath_support in test_io. (GH-21640)
https://github.com/python/cpython/commit/67987acd5dc9776f55f4e139e2b3d9e7a6434d9f
msg374409 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-27 18:07
Automatic backport does not work due to changes in the test.support module. Victor, do you mind to backport PR 21640 to 3.9 and 3.8?
msg374598 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-07-30 05:36
The 'master' branch bot is working again, the 3.9 branch is still broken, and the 3.8 branch seems, as yet, unaffected.
msg375598 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-08-18 08:15
As much as I wish I had the skills to do the cherry picking - I am not going to touch this.

The AIX bots for 3.9 branch continue to report broken for test_io (ENV change) - as they still wait for the backport for that branch!
msg376497 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-07 14:17
> Automatic backport does not work due to changes in the test.support module. Victor, do you mind to backport PR 21640 to 3.9 and 3.8?

Why don't you want to do backport yourself? It's a trivial change, it only changes a single line.

Anyway, I created PR 22133.
msg376499 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-07 14:46
New changeset c73ee5acc96b4bbd6885156883b224b8cc3e470c by Victor Stinner in branch '3.9':
bpo-41401: Fix test_fspath_support in test_io. (GH-21640) (GH-22133)
https://github.com/python/cpython/commit/c73ee5acc96b4bbd6885156883b224b8cc3e470c
msg376502 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-07 15:27
New changeset 84105cbaa395923e3584a87d67ccce72c8420bb4 by Victor Stinner in branch '3.8':
bpo-41401: Fix test_fspath_support in test_io. (GH-21640) (GH-22133) (GH-22135)
https://github.com/python/cpython/commit/84105cbaa395923e3584a87d67ccce72c8420bb4
msg376503 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-07 15:28
It should now be fixed. Thanks.
History
Date User Action Args
2020-12-31 12:45:33serhiy.storchakasetpull_requests: + pull_request22864
2020-09-07 15:28:45vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg376503

stage: patch review -> resolved
2020-09-07 15:27:25vstinnersetmessages: + msg376502
2020-09-07 14:47:10vstinnersetpull_requests: + pull_request21217
2020-09-07 14:46:13vstinnersetmessages: + msg376499
2020-09-07 14:17:28vstinnersetmessages: + msg376497
2020-09-07 14:16:31vstinnersetnosy: + vstinner
pull_requests: + pull_request21215
2020-09-05 13:45:09David.Edelsohnsetnosy: + David.Edelsohn
2020-09-05 13:26:54skrahlinkissue41717 superseder
2020-08-18 08:15:25Michael.Feltsetmessages: + msg375598
2020-08-03 23:06:06vstinnersetnosy: - vstinner
2020-07-30 05:36:36Michael.Feltsetmessages: + msg374598
2020-07-27 18:07:46serhiy.storchakasetnosy: + vstinner
messages: + msg374409
2020-07-27 17:58:46serhiy.storchakasetmessages: + msg374406
2020-07-27 14:26:12Michael.Feltsetmessages: + msg374379
2020-07-27 14:16:24serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request20781
2020-07-27 14:14:16serhiy.storchakasetmessages: + msg374377
2020-07-26 20:56:14Michael.Feltsetmessages: + msg374343
2020-07-26 14:32:02serhiy.storchakasetmessages: + msg374316
2020-07-26 14:15:16xtreaksetnosy: + serhiy.storchaka
2020-07-26 14:07:03Michael.Feltcreate