classification
Title: test_os.test_copy_file_range() killed by SIGSYS (12) on FreeBSD CURRENT buildbot
Type: Stage: resolved
Components: Tests Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: koobs, pablogsal, vstinner
Priority: normal Keywords:

Created on 2019-08-27 06:59 by vstinner, last changed 2019-08-28 23:43 by vstinner. This issue is now closed.

Messages (13)
msg350609 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 06:59
https://buildbot.python.org/all/#/builders/168/builds/1356

test_nop (test.test_os.FSEncodingTests) ... ok
test_access (test.test_os.FileTests) ... ok
test_closerange (test.test_os.FileTests) ... ok
test_copy_file_range (test.test_os.FileTests) ... *** Signal 12
Stop.
make: stopped in /usr/home/buildbot/python/3.x.koobs-freebsd-current/build
program finished with exit code 1
elapsedTime=1562.592656

man signal says:

"12	   SIGSYS	create core image    non-existent system call invoked"

I bet that the "non-existent system call" is: copy_file_range. The configure script says:

"checking for copy_file_range... yes"


Se also bpo-37711: "regrtest: re-run failed tests in subprocesses".
msg350618 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 09:27
Oh, FreeBSD, what have you done now.....
msg350620 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 09:31
@koobs, did you changed recently the buildbot?
msg350621 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-08-27 09:33
@Pablo Upgrade to latest CURRENT revision a few hours ago. Some system software is still updating. I'll push a rebuild on BB once complete. Stand by :)
msg350622 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-08-27 09:36
Possibly related:

https://reviews.freebsd.org/D20584 - Add a linux compatible copy_file_range(2) syscall

https://lists.freebsd.org/pipermail/freebsd-current/2019-July/073747.html
msg350624 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-08-27 09:40
See Also: #26826
msg350627 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 10:14
It would be better if the FreeBSD kernel would return ENOSYS error code rather than killing the process with SIGSYS :-)
msg350628 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-08-27 11:46
Hmm... a test checkout of master/default on the host in question has test_copy_file_range{_*} passing:

[user@CURRENT-amd64:/usr/home/user/repos/cpython] ./python -m test -vvv test_os |grep range
...
test_copy_file_range (test.test_os.FileTests) ... ok
test_copy_file_range_invalid_values (test.test_os.FileTests) ... ok
test_copy_file_range_offset (test.test_os.FileTests) ... ok
...
Ran 280 tests in 2.503s

OK (skipped=45)
msg350629 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-27 11:54
> https://buildbot.python.org/all/#/builders/168/builds/1356

Just to confirm: test_os was run twice and was killed twice by signal 12.

0:00:04 load avg: 3.45 [  8/419/1] test_os crashed (Exit code -12)
...
Re-running failed tests in verbose mode
Re-running test_os in verbose mode
...
test_copy_file_range (test.test_os.FileTests) ... *** Signal 12
msg350631 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-08-27 12:20
Ran rebuild (in the BB UI) on that failed build (#1356), and test_os now passes:

0:22:52 load avg: 8.38 [339/419] test_os passed

Victor independently logged into the host and ran a from scratch build/test and test_os passes for him too (verifying my from scratch test results)

Only thing I can think of is a cached/inconsistent BB build environment after the host FreeBSD update, that cleared itself after the first build
msg350632 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 12:59
Could this be that the libc was outdated?
msg350650 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-08-28 02:57
Possibly, but unlikely as the system had been freshly installed and rebooted before the worker was restarted.

The worker processes, however, had not been stopped prior to, or during the upgrade, so it's possible that the worker had an inconsistent build state that had resumed, or used before-upgrade information, code or objects during tests
msg350695 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-28 23:43
AMD64 FreeBSD CURRENT Shared 3.x is back to green. I close the issue.

https://buildbot.python.org/all/#/builders/168/builds/1356 likely failed because kkobs upgrade the FreeBSD CURRENT VM while the buildbot job was running (inconsistency between *running* libc and kernel, I guess).
History
Date User Action Args
2019-08-28 23:43:44vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg350695

stage: resolved
2019-08-28 02:57:36koobssetmessages: + msg350650
2019-08-27 12:59:01pablogsalsetmessages: + msg350632
2019-08-27 12:20:17koobssetmessages: + msg350631
2019-08-27 11:54:32vstinnersetmessages: + msg350629
2019-08-27 11:46:03koobssetmessages: + msg350628
2019-08-27 10:14:39vstinnersetmessages: + msg350627
2019-08-27 09:40:39koobssetmessages: + msg350624
2019-08-27 09:36:28koobssetmessages: + msg350622
2019-08-27 09:33:38koobssetmessages: + msg350621
2019-08-27 09:31:26pablogsalsetmessages: + msg350620
2019-08-27 09:30:58pablogsalsetnosy: + koobs
2019-08-27 09:27:07pablogsalsetmessages: + msg350618
2019-08-27 06:59:15vstinnercreate