classification
Title: Allow regrtest to receive a file with test (and subtests) to ignore
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: pablogsal, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2019-08-27 00:03 by pablogsal, last changed 2019-11-20 01:02 by pablogsal.

Pull Requests
URL Status Linked Edit
PR 16989 merged pablogsal, 2019-10-30 00:02
Messages (9)
msg350584 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 00:03
When building Python in some uncommon platforms (I am looking at you Solaris and AIX) there are some known tests that will fail. Right now, regrtest has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter.

The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept.

The problem with -x is that is not easy to ignore just a subtests that fail and the whole tests needs to be ignored.

So I suggest to add a new command line option similar to --matchfile but the other way around. Another possibility is allowing to reverse the meaning of the matchfile argument, but I find that a bit more confusing.
msg350601 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 06:27
I'm fine with adding an --excludefile=filename option to regrtest. Do you want to try to implement it? A test can be added in test_regrtest.
msg350606 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-08-27 06:50
Tests that fail on some platforms can be marked to skip in code.

For example:

    @unittest.skipIf(sys.platform.startswith('aix'),
                     'bpo-29972: broken test on AIX')
    def test_strcoll_with_diacritic(self):
        self.assertLess(locale.strcoll('à', 'b'), 0)
msg350607 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 06:54
> Tests that fail on some platforms can be marked to skip in code.

We provide best-effort support for AIX, but I consider that Solaris is no longer supported, and so I would prefer to avoid code specific to Soliars in the Python source code. Except if Pablo wants to provide again best-effort support for Solaris?
https://pythondev.readthedocs.io/platforms.html
msg350617 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 08:48
> Tests that fail on some platforms can be marked to skip in code.

One of the use cases of this feature is for people to package Python at distributions or companies and want to run the test suite without modifying the code or applying patches.
msg350626 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 10:13
FYI in the past, Fedora and RHEL packages skipped multiple tests using downstream patches. Hopefully, we managed to reenable almost all tests.

Currently in Fedora, the python3 package use regrtest -x to skip some tests:

  # Run the upstream test suite
  # test_gdb skipped on s390x:
  #   https://bugzilla.redhat.com/show_bug.cgi?id=1678277
  # test_gdb skipped everywhere:
  #   https://bugzilla.redhat.com/show_bug.cgi?id=1734327
  # test_distutils
  #   distutils.tests.test_bdist_rpm tests fail when bootstraping the Python
  #   package: rpmbuild requires /usr/bin/pythonX.Y to be installed
  LD_LIBRARY_PATH=$ConfDir $ConfDir/python -m test.regrtest \
    -wW --slowest -j0 \
    %if %{with bootstrap}
    -x test_distutils \
    %endif
    -x test_gdb \
    %ifarch %{mips64}
    -x test_ctypes \
    %endif

https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec

We could only skip some test methods, but it's simpler for us to skip a whole test file.
msg357006 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-19 23:46
New changeset e0cd8aa70a3ce19c3d3712568940aa0cbd9aa97b by Pablo Galindo in branch 'master':
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
https://github.com/python/cpython/commit/e0cd8aa70a3ce19c3d3712568940aa0cbd9aa97b
msg357015 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 01:01
I'm trying to keep regrtest in sync between 3.7, 3.8 and master branches. Maybe backport this change to 3.7 and 3.8 branches once buildbots validated the change? I'm keeping them in sync mostly to make bugfixes easier, but also to make my life easier when I debug issues in 3.7 or 3.8 branch.
msg357016 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-20 01:02
Ok, I reopen until the backports
History
Date User Action Args
2019-11-20 01:02:51pablogsalsetstatus: closed -> open
resolution: fixed ->
messages: + msg357016
2019-11-20 01:01:39vstinnersetmessages: + msg357015
2019-11-19 23:47:30pablogsalsetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: - Python 3.8
2019-11-19 23:46:57pablogsalsetmessages: + msg357006
2019-10-30 00:02:34pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request16515
2019-08-27 10:13:55vstinnersetmessages: + msg350626
2019-08-27 08:48:53pablogsalsetmessages: + msg350617
2019-08-27 06:54:25vstinnersetmessages: + msg350607
2019-08-27 06:50:23serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg350606
2019-08-27 06:27:58vstinnersetmessages: + msg350601
2019-08-27 00:03:54pablogsalcreate