Title: Allow regrtest to receive a file with test (and subtests) to ignore
Type: enhancement Stage:
Components: Tests Versions: Python 3.9, Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: pablogsal, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2019-08-27 00:03 by pablogsal, last changed 2019-08-27 10:13 by vstinner.

Messages (6)
msg350584 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 00:03
When building Python in some uncommon platforms (I am looking at you Solaris and AIX) there are some known tests that will fail. Right now, regrtest has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter.

The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept.

The problem with -x is that is not easy to ignore just a subtests that fail and the whole tests needs to be ignored.

So I suggest to add a new command line option similar to --matchfile but the other way around. Another possibility is allowing to reverse the meaning of the matchfile argument, but I find that a bit more confusing.
msg350601 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 06:27
I'm fine with adding an --excludefile=filename option to regrtest. Do you want to try to implement it? A test can be added in test_regrtest.
msg350606 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-08-27 06:50
Tests that fail on some platforms can be marked to skip in code.

For example:

                     'bpo-29972: broken test on AIX')
    def test_strcoll_with_diacritic(self):
        self.assertLess(locale.strcoll('à', 'b'), 0)
msg350607 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 06:54
> Tests that fail on some platforms can be marked to skip in code.

We provide best-effort support for AIX, but I consider that Solaris is no longer supported, and so I would prefer to avoid code specific to Soliars in the Python source code. Except if Pablo wants to provide again best-effort support for Solaris?
msg350617 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 08:48
> Tests that fail on some platforms can be marked to skip in code.

One of the use cases of this feature is for people to package Python at distributions or companies and want to run the test suite without modifying the code or applying patches.
msg350626 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 10:13
FYI in the past, Fedora and RHEL packages skipped multiple tests using downstream patches. Hopefully, we managed to reenable almost all tests.

Currently in Fedora, the python3 package use regrtest -x to skip some tests:

  # Run the upstream test suite
  # test_gdb skipped on s390x:
  # test_gdb skipped everywhere:
  # test_distutils
  #   distutils.tests.test_bdist_rpm tests fail when bootstraping the Python
  #   package: rpmbuild requires /usr/bin/pythonX.Y to be installed
  LD_LIBRARY_PATH=$ConfDir $ConfDir/python -m test.regrtest \
    -wW --slowest -j0 \
    %if %{with bootstrap}
    -x test_distutils \
    -x test_gdb \
    %ifarch %{mips64}
    -x test_ctypes \

We could only skip some test methods, but it's simpler for us to skip a whole test file.
Date User Action Args
2019-08-27 10:13:55vstinnersetmessages: + msg350626
2019-08-27 08:48:53pablogsalsetmessages: + msg350617
2019-08-27 06:54:25vstinnersetmessages: + msg350607
2019-08-27 06:50:23serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg350606
2019-08-27 06:27:58vstinnersetmessages: + msg350601
2019-08-27 00:03:54pablogsalcreate