classification
Title: test_regrtest leaves a test_python_* directory in TEMPDIR
Type: behavior Stage: patch review
Components: Tests Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2017-12-08 08:49 by xdegaye, last changed 2017-12-11 13:17 by vstinner.

Pull Requests
URL Status Linked Edit
PR 4794 merged vstinner, 2017-12-11 12:32
PR 4795 merged python-dev, 2017-12-11 12:57
Messages (13)
msg307837 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-08 08:49
After running test_regrtest in the source tree on linux, the build/ subdirectory (i.e. test.libregrtest.main.TEMPDIR) contains a new test_python_* directory that contains a core file when the core file size is unlimited.

I did not test on 3.6.
msg307855 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-08 16:54
> After running test_regrtest in the source tree on linux, the build/ subdirectory (i.e. test.libregrtest.main.TEMPDIR) contains a new test_python_* directory that contains a core file when the core file size is unlimited.

Can you please explain how to reproduce the bug? (Which commands should I type?)

What is your /proc/sys/kernel/core_pattern?

On my Fedora 27, /proc/sys/kernel/core_pattern is "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e".
msg307885 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-09 09:07
> Can you please explain how to reproduce the bug? (Which commands should I type?)

./python -m test test_regrtest

> What is your /proc/sys/kernel/core_pattern?

$ cat /proc/sys/kernel/core_pattern
core-%e.%s
$ ulimit -c
unlimited

FWIW on archlinux (my platform) coredumps are handled by systemd and to enable coredumps the file /etc/sysctl.d/50-coredump.conf has to be updated to contain:

kernel.core_pattern=core-%e.%s
msg307886 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-09 09:30
Oh it seems your Fedora 27 also uses systemd and uses the same configuration as archlinux, see https://wiki.archlinux.org/index.php/Core_dump. In that case according to this wiki your core dumps go to /var/lib/systemd/coredump and this may explain the different behaviors between your system and mine.

My setup (updating the file /etc/sysctl.d/50-coredump.conf) was the documented archlinux practice 4 years ago when systemd used to store the coredumps in log files (very annoying). This setup is still working although it is not documented in this wiki (maybe somewhere else).

FWIW I have kept the notes made when configuring archlinux at that time, they are:

disable core dumps managed by systemd-coredumpctl(1) - see also man pages for sysctl.d and sysctl
/etc/sysctl.d/50-coredump.conf                        # same file name as in /usr/lib/sysctl.d/
    kernel.core_pattern=core-%e.%s
echo "core-%e.%s" > /proc/sys/kernel/core_pattern     # or reboot

# per user
ulimit -c unlimited
msg308032 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-11 12:05
I'm unable to reproduce the issue on Fedora 27:

vstinner@apu$ cat /proc/sys/kernel/core_pattern 
core-%e.%s
vstinner@apu$ ulimit -c
unlimited
vstinner@apu$ ./python -m test test_regrtest
Run tests sequentially
0:00:00 load avg: 0.70 [1/1] test_regrtest
1 test OK.

Total duration: 14 sec
Tests result: SUCCESS


While core dump works as expected:

vstinner@apu$ ./python
>>> import faulthandler
>>> faulthandler._sigsegv()
Segmentation fault (core dumped)
vstinner@apu$ ls core*
core-python.11.32456
msg308033 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-11 12:14
Ah! I misunderstood the bug report. I was looking for a ENV_FAILED failure, but no, regrtest fails to remove its temporary directory but no warning is emitted in this case.

vstinner@apu$ ls -d build/test_python_*; ./python -m test test_regrtest -m test.test_regrtest.ArgsTestCase.test_crashed; ls -d build/test_python_*
ls: impossible d'accéder à 'build/test_python_*': No such file or directory

Run tests sequentially
0:00:00 load avg: 0.42 [1/1] test_regrtest
1 test OK.

Total duration: 537 ms
Tests result: SUCCESS

build/test_python_816


So running test_regrtest creates build/test_python_816/ but fails to remove it because it contains a core dump: core-python.11.816.

* No core dump must be written, Python is supposed to disable that
* regrtest must emit a warning if it fails to remove the temporary directory, maybe also fail with an error? (non-zero exit code)
msg308034 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-11 12:19
What is now the content of the 'build/' subdirectory of the source tree ?
msg308035 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-11 12:22
> Ah! I misunderstood the bug report. I was looking for a ENV_FAILED failure, but no, regrtest fails to remove its temporary directory but no warning is emitted in this case.

Yes :-)
msg308036 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-11 12:25
> So running test_regrtest creates build/test_python_816/ but fails to remove it because it contains a core dump: core-python.11.816.

Not sure that it is because it contains a core file. Maybe it just does not remove it because the test crashed ?
msg308037 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-11 12:36
> Maybe it just does not remove it because the test crashed ?

Ah, you're right :-)
msg308038 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-11 12:37
> * No core dump must be written, Python is supposed to disable that

My bad. It's an obvious bug that was missed before because regrtest creates a temporary directory to run tests and then remove it.

I wrote the PR 4794 to fix faulthandler.
msg308039 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-11 12:57
New changeset 48d4dd974f0c8d47c54990eedd322b96b19c60ec by Victor Stinner in branch 'master':
bpo-32252: Fix faulthandler_suppress_crash_report() (#4794)
https://github.com/python/cpython/commit/48d4dd974f0c8d47c54990eedd322b96b19c60ec
msg308041 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-11 13:17
New changeset 71d8f36eb450cdbc4b5397e25f6f3f5d676aca79 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
bpo-32252: Fix faulthandler_suppress_crash_report() (GH-4794) (#4795)
https://github.com/python/cpython/commit/71d8f36eb450cdbc4b5397e25f6f3f5d676aca79
History
Date User Action Args
2017-12-11 13:17:09vstinnersetmessages: + msg308041
2017-12-11 12:57:35python-devsetpull_requests: + pull_request4694
2017-12-11 12:57:15vstinnersetmessages: + msg308039
2017-12-11 12:37:47vstinnersetmessages: + msg308038
2017-12-11 12:36:36vstinnersetmessages: + msg308037
2017-12-11 12:32:14vstinnersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request4693
2017-12-11 12:25:48xdegayesetmessages: + msg308036
2017-12-11 12:22:02xdegayesetmessages: + msg308035
2017-12-11 12:19:41xdegayesetmessages: + msg308034
2017-12-11 12:14:09vstinnersetmessages: + msg308033
2017-12-11 12:05:40vstinnersetmessages: + msg308032
2017-12-09 09:30:52xdegayesetmessages: + msg307886
2017-12-09 09:07:59xdegayesetmessages: + msg307885
2017-12-08 16:54:37vstinnersetmessages: + msg307855
2017-12-08 08:49:13xdegayecreate