test_regrtest: test_list_cases() fails on x86 Gentoo Installed with X 3.x #81540
Comments
|
The root issue is likely related to "RuntimeWarning: tests may fail, unable to create temporary directory '/buildbot/tmp/tmpdir/test_python_28144': [Errno 17] File exists". I don't know if a previous buildbot run forgot to remove this directory, or if it's a race condition in test_regrtest? https://buildbot.python.org/all/#/builders/103/builds/2679 FAIL: test_list_cases (test.test_regrtest.ArgsTestCase) Traceback (most recent call last):
File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_regrtest.py", line 915, in test_list_cases
self.assertEqual(output.splitlines(), all_methods)
AssertionError: Lists differ: ['tes[73 chars]hod2', "/buildbot/buildarea/cpython/3.x.ware-g[307 chars]th:'] != ['tes[73 chars]hod2']First list contains 2 additional elements. ['test_regrtest_noop8.Tests.test_method1',
+ 'test_regrtest_noop8.Tests.test_method2']
---------------------------------------------------------------------- |
|
Related change: commit 3c93153 |
|
Another failure: https://buildbot.python.org/all/#/builders/103/builds/2688/steps/6/logs/stdio ====================================================================== Traceback (most recent call last):
File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_regrtest.py", line 915, in test_list_cases
self.assertEqual(output.splitlines(), all_methods)
AssertionError: Lists differ: ['tes[73 chars]hod2', "/buildbot/buildarea/cpython/3.x.ware-g[305 chars]th:'] != ['tes[73 chars]hod2']
First list contains 2 additional elements.
First extra element 2:
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/support/__init__.py:1058: RuntimeWarning: tests may fail, unable to create temporary directory '/buildbot/tmp/tmpdir/test_python_5670': [Errno 17] File exists: '/buildbot/tmp/tmpdir/test_python_5670'"
['test_regrtest_noop8.Tests.test_method1',
- 'test_regrtest_noop8.Tests.test_method2',
? ^
+ 'test_regrtest_noop8.Tests.test_method2']
? ^
- '/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/support/__init__.py:1058: '
- 'RuntimeWarning: tests may fail, unable to create temporary directory '
- "'/buildbot/tmp/tmpdir/test_python_5670': [Errno 17] File exists: "
- "'/buildbot/tmp/tmpdir/test_python_5670'",
- ' with temp_dir(path=name, quiet=quiet) as temp_path:']Ran 73 tests in 35.451s |
|
I created python/buildmaster-config#98 to cleanup tests in buildbots. |
|
I deployed the new buildbot configuration. New builds should run "make cleanup" (Unix) or "python -m test --cleanup" (Windows) on Python 2.7, 3.7, 3.8 and master, but not on custom branches. |
|
cleantest was run successfully on the 4 Fedora workers: 3.x: https://buildbot.python.org/all/#/builders/185/builds/641 |
|
cleantest was also run on Windows buildbots. Example a Windows 2.7 buildbot: D:\buildarea\2.7.ware-win81-release.nondebug\build>"D:\buildarea\2.7.ware-win81-release.nondebug\build\PCbuild\amd64\python" -Wd -3 -E -tt "D:\buildarea\2.7.ware-win81-release.nondebug\build\PCbuild\..\Lib\test\regrtest.py" -uall -rwW --slowest -j2 -j4 --cleanup So a previous test job leaked "test_python_4824" directory. |
|
I am not 100% sure that my change will fix the following warning on x86 Gentoo Installed with X 3.x, but I'm quite confident :-) RuntimeWarning: tests may fail, unable to create temporary directory '/buildbot/tmp/tmpdir/test_python_28144': [Errno 17] File exists: '/buildbot/tmp/tmpdir/test_python_28144'" I close the issue. Reopen it if the bug strikes back. |
|
The issue strikes back: |
Not good. This build was run after I deployed the new buildbot configuraiton, with the new "cleantest" step, but it has no "cleantest" step. Maybe this job was scheduled before the new configuration was deployed? I rebuilt this build to see if it gets "cleantest" step: Note: I'm not surprised that the issue strikes back. It happens all the time with buildbots... |
|
Oh I see, this specific buildbot uses a different buildbot configuration that I forgot to patch. It should now be fixed as well: New build which should use this configuration: |
|
I wasn't sure if cleantest was needed, since I expected that buildbot is stateless and always run a build in a clean environment... Well... no. It's needed. My new step ran and it removes 4146 old "test_python_xxx" directories... That's pretty big knowning that Linux pids are limited to 32768: 4146 is already 13% of the available ids, which makes the issue quite likely... Cleanup /buildbot/tmp/tmpdir directory |
|
x86 Gentoo Installed with X 3.x is fixed as well. I close again the issue. |
|
Oh, this feature has an issue on buildbots. For the 2 workers which allows more than 1 job in parallel, cleantest removes working directories of other running tests... "x86 Gentoo Installed with X 3.7" is fighting with "x86 Gentoo Installed with X 3.8" for example: when a 3.8 job started, it removed working directories of 2 tests of the 3.7 job. This buildbot uses a single temporary directory for all Python branches: /buildbot/tmp/tmpdir. I'm not sure how to fix this issue:
|
Another idea is to change the name of temporary directories to add a different prefix per Python branch. Like use "test_python_39_<pid>" for Python 3.9. But I'm not sure how custom jobs come into the play here. Python is not aware of "custom", it has a fixed version and Git branch. |
|
Until a solution is found, I disabled cleantest to repair *again* the Gentoo buildbot worker: |
|
The current status is that regrtest has a new --cleanup command but it's not used. regrtest should not leak temporary files anymore. The remaining issue is when regrtest re-runs tests sequentially and a test does crash the regrtest main process. In this case, the temporary directory is not removed. I created bpo-37711 "regrtest: re-run failed tests in subprocesses" for this case. The initial issue has been fixed, so I close the issue. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: