classification
Title: Random failures when running test suite in parallel (-m test -j0) caused by test_regrtest
Type: Stage:
Components: Tests Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, martin.panter, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2016-02-05 09:15 by vstinner, last changed 2016-04-01 19:48 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
test_regrtest_tmpdir.patch vstinner, 2016-03-11 11:55 review
test_regrtest_tmpdir-2.patch vstinner, 2016-03-15 22:57 review
Messages (12)
msg259647 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-05 09:15
test_regrtest creates temporary test files called test_regrtest_pid_xxx.py in Lib/test/. The problem is that some tests like test___all__ and test_zipfile haves test relying on the list of Lib/test/test_*.py.

When tests are run in parallel, test_regrtest can creates temporary test_regrtest_pid_xxx.py files, test_zipfile sees them, test_regrtest removes them, and then test_zipfiles fails.

The best would be to write these temporary files into a temporary directory, but I failed to fix Lib/test/regrtest.py to load tests from a different directory. In theory, Python 3 supports packages with files splitted into multiple directories, in practice it doesn't seem to work :-p

Maybe test_regrtest should use a test submodule like Lib/test/temp_regrtest/ ?

test_regrtest started to create temporary test_xxx.py files since issue #25220. (Other changes to test_regrtest: issues #18174, #22806, #25260, #25306,  #25369, #25373, #25694).

======================================================================
ERROR: test_all (test.test___all__.AllTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test___all__.py", line 102, in test_all
    with open(path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/haypo/prog/python/default/Lib/test/test_regrtest_25743_noop20.py'

----------------------------------------------------------------------
msg259649 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-02-05 09:36
The directory with test files can be read-only. test_regrtest should use temporary directory.
msg259650 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-05 09:38
> The directory with test files can be read-only.

test_regrtest skips tests which requires to write in Lib/test/ if the write fails.

        except PermissionError as exc:
            if not sysconfig.is_python_build():
                self.skipTest("cannot write %s: %s" % (path, exc))

> test_regrtest should use temporary directory.

Yes! That's the purpose of this issue!

But I was unable to find a technical solution to implement this. I need to try harder :-)
msg260607 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-21 11:21
You say you couldn’t get the test package to be split into multiple directories. What did you try? As far as I can see, we need a way for test_regrtest to invoke the child process with some flag to say “add this extra directory to test.__path__”.
msg260609 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-21 11:32
Another idea is to make a loader script that loads the “test” module, patches its __path__, and then runs test.__main__ or whatever as usual.
msg261563 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-11 11:55
Oh ok, I found a bug in the setup_tests() function of libregrtest: issue #26538.

Attached patch fixes this issue, it requires the patch of the issue issue #26538.
msg261833 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-15 22:57
Hum, test_doctest, test_zipfile and test_pyclbr fail if test is converted to a package (if Lib/test/__init__.py is removed).

Attached patch fixes test_zipfile.

I created the issue #26569 for test_pyclbr failure.
msg262430 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-25 12:01
I created the issue #26641 for doctest (test_doctest).
msg262640 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-30 00:43
New changeset f7ddd72b70ea by Victor Stinner in branch 'default':
Issue #26295: When using "python3 -m test --testdir=TESTDIR", regrtest doesn't
https://hg.python.org/cpython/rev/f7ddd72b70ea

New changeset c83349c129c9 by Victor Stinner in branch 'default':
Issue #26295: test_regrtest now uses a temporary directory
https://hg.python.org/cpython/rev/c83349c129c9
msg262650 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-30 06:50
New changeset e29f12ed4444 by Victor Stinner in branch 'default':
Issue #26295: Fix test_regrtest.test_tools_buildbot_test()
https://hg.python.org/cpython/rev/e29f12ed4444
msg262651 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-30 06:51
New changeset e47e00723e5d by Victor Stinner in branch 'default':
Issue #26295: Enhanc test_regrtest.test_tools_script_run_tests()
https://hg.python.org/cpython/rev/e47e00723e5d
msg262763 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-04-01 19:48
Finally, I fixed the issue differently: test_regrtest now creates a temporary directory and uses --testdir command line parameter.

Before, using --testdir didn't work because regrtest always added "test." prefix to module names (ex: test_os => test.test_os). It isn't the case anymore when --testdir is used.

Buildbots are green, so I close the issue.
History
Date User Action Args
2016-04-01 19:48:50vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg262763
2016-03-30 06:51:22python-devsetmessages: + msg262651
2016-03-30 06:50:19python-devsetmessages: + msg262650
2016-03-30 00:43:16python-devsetnosy: + python-dev
messages: + msg262640
2016-03-25 12:01:10vstinnersetmessages: + msg262430
2016-03-15 22:57:35vstinnersetfiles: + test_regrtest_tmpdir-2.patch

messages: + msg261833
2016-03-11 11:55:23vstinnersetfiles: + test_regrtest_tmpdir.patch
keywords: + patch
messages: + msg261563
2016-02-21 11:32:58martin.pantersetmessages: + msg260609
2016-02-21 11:21:14martin.pantersetnosy: + martin.panter
messages: + msg260607
2016-02-05 09:38:45vstinnersetmessages: + msg259650
2016-02-05 09:36:49serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg259649
2016-02-05 09:15:07vstinnercreate