classification
Title: Test code coverage (C) job of Travis CI fails on test_distutils which creates _configtest.gcno file
Type: Stage: patch review
Components: Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ammar2, brett.cannon, corona10, p-ganssle, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-04-09 12:52 by vstinner, last changed 2020-06-17 08:03 by vstinner.

Pull Requests
URL Status Linked Edit
PR 19460 open shihai1991, 2020-04-10 16:49
Messages (15)
msg366047 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-09 12:52
Example:
https://travis-ci.org/github/python/cpython/jobs/672698615

Output:
===========================================
0:09:01 load avg: 3.96 [104/421/1] test_distutils failed -- running: test_multiprocessing_forkserver (1 min 13 sec), test_unparse (6 min 21 sec)

Warning -- files was modified by test_distutils

  Before: []

  After:  ['_configtest.gcno'] 

test test_distutils failed -- Traceback (most recent call last):

  File "/home/travis/build/python/cpython/Lib/distutils/tests/test_build_ext.py", line 112, in test_build_ext

    assert_python_ok('-c', code)

  File "/home/travis/build/python/cpython/Lib/test/support/script_helper.py", line 156, in assert_python_ok

    return _assert_python(True, *args, **env_vars)

  File "/home/travis/build/python/cpython/Lib/test/support/script_helper.py", line 142, in _assert_python

    res.fail(cmd_line)

  File "/home/travis/build/python/cpython/Lib/test/support/script_helper.py", line 70, in fail

    raise AssertionError("Process return code is %d\n"

AssertionError: Process return code is 1

command line: ['/home/travis/build/python/cpython/python', '-X', 'faulthandler', '-I', '-c', "\ntmp_dir = '/tmp/tmp3y4hpawx'\n\nimport sys\nimport unittest\nfrom test import support\n\nsys.path.insert(0, tmp_dir)\nimport xx\n\nclass Tests(unittest.TestCase):\n    def test_xx(self):\n        for attr in ('error', 'foo', 'new', 'roj'):\n            self.assertTrue(hasattr(xx, attr))\n\n        self.assertEqual(xx.foo(2, 5), 7)\n        self.assertEqual(xx.foo(13,15), 28)\n        self.assertEqual(xx.new().demo(), None)\n        if support.HAVE_DOCSTRINGS:\n            doc = 'This is a template module just for instruction.'\n            self.assertEqual(xx.__doc__, doc)\n        self.assertIsInstance(xx.Null(), xx.Null)\n        self.assertIsInstance(xx.Str(), xx.Str)\n\n\nunittest.main()\n"]

stdout:

---

---

stderr:

---

Traceback (most recent call last):

  File "<string>", line 9, in <module>

ImportError: /tmp/tmp3y4hpawx/xx.cpython-39-x86_64-linux-gnu.so: undefined symbol: __gcov_merge_add

---

===========================================

Python is built GCC with -ftest-coverage option. But it seems like this option is "leaked" to C flags used to build third party extensions in distutils.

Maybe CFLAGS_NODIST should be used instead of CFLAGS.

The Travis CI job runs these commands:
---
./configure
xvfb-run make -j4 coverage-report
make pythoninfo
bash <(curl -s https://codecov.io/bash) -y .github/codecov.yml
---
msg366140 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-04-10 16:55
> Maybe CFLAGS_NODIST should be used instead of CFLAGS.
Looks like it worked.

> Python is built GCC with -ftest-coverage option.
More exact description is: Python is built GCC with --coverage option, --coverage option including -ftest-coverage option.
msg366141 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-04-10 16:58
> Looks like it worked.
Oh, sorry, I checked the wrong gate.
msg366198 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-04-11 10:41
After `CFLAGS` replcaced by `CFLAGS_NODIST`, some extension module built failed, some info like:
*** WARNING: renaming "_struct" since importing it failed: build/lib.linux-x86_64-3.9/_struct.cpython-39-x86_64-linux-gnu.so: undefined symbol: __gcov_merge_add

the possible reason:
building '_struct' extension
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -O0 -pg --coverage -I./Include/internal -I./Include -I. -I/usr/local/include -I/temp/shihai/cpython/Include -I/temp/shihai/cpython -c /temp/shihai/cpython/Modules/_struct.c -o build/temp.linux-x86_64-3.9/temp/shihai/cpython/Modules/_struct.o
gcc -pthread -shared (xxxxxxxxxxlacking --coverage in here xxxxxxxxx) build/temp.linux-x86_64-3.9/temp/shihai/cpython/Modules/_struct.o -L/usr/local/lib -o build/lib.linux-x86_64-3.9/_struct.cpython-39-x86_64-linux-gnu.so
msg366199 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-04-11 10:51
FWIW, gcc Instrumentation Options in https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html
msg366679 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-17 21:12
I understand that with PR 19460, code coverage is not longer run on C extension modules of the stdlib. I'm not using C code coverage, so I don't know how it's supposed to be used or analyzed.

Hai: if you want to move on this issue, you have to find who uses this CI and how it should behave.

The other option is to only fix test_distutils.
msg366693 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-04-18 05:24
> Hai: if you want to move on this issue, you have to find who uses this CI and how it should behave.

Copy that. I found PR7773 updated the travis' coverage ci and a discussion issue in https://github.com/python/core-workflow/issues/2, so I add brett and ammar to the noisy list, MAYBE they know the historical background.
msg366867 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-04-20 19:06
The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it.
msg366921 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-04-21 15:23
> The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it.

Hm, code coverage of C code including two parts:
1) coverage of inner c code;
2) extension modules built by distutils;

If I understand correct, we force on 1) is good enough, no?
msg366924 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-21 15:41
Brett:
> The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it.

If there is no user of code coverage *on pull requests*, maybe the simplest fix is to remove this job.

There is also Codecov service running somehow on Python branches through GitHub integration.
msg368869 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-14 21:48
It seems like results at published at: https://codecov.io/gh/python/cpython/

It seems like there is coverage on both C and Python code. I never used this service.
msg368926 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-05-15 05:01
> It seems like there is coverage on both C and Python code.

Great, MAYBE we can use this coverage result to improve our testcases.
msg371665 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-16 15:06
See also bpo-40993: "Don't run Python and C coverage jobs of Travis CI on pull requests".
msg371716 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 08:00
See also Coverage report in Github CI for standard library discussion:
https://discuss.python.org/t/coverage-report-in-github-ci-for-standard-library/2836
msg371717 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 08:03
Another issue: the "Test code coverage (C)" job on Travis CI times out after 50 minutes:
https://travis-ci.org/github/python/cpython/jobs/699170857

0:44:44 load avg: 7.76 running: test_multiprocessing_spawn (11 min 38 sec), test_concurrent_futures (9 min 33 sec), test_unparse (25 min 7 sec), test_capi (1 min)
0:45:14 load avg: 7.91 running: test_multiprocessing_spawn (12 min 8 sec), test_concurrent_futures (10 min 3 sec), test_unparse (25 min 37 sec), test_capi (1 min 30 sec)
0:45:44 load avg: 8.21 running: test_multiprocessing_spawn (12 min 38 sec), test_concurrent_futures (10 min 33 sec), test_unparse (26 min 7 sec), test_capi (2 min)
0:46:14 load avg: 7.33 running: test_multiprocessing_spawn (13 min 8 sec), test_concurrent_futures (11 min 3 sec), test_unparse (26 min 37 sec), test_capi (2 min 30 sec)

(...)

The job exceeded the maximum time limit for jobs, and has been terminated.
History
Date User Action Args
2020-06-17 08:03:00vstinnersetmessages: + msg371717
2020-06-17 08:00:39vstinnersetmessages: + msg371716
2020-06-16 15:06:19vstinnersetmessages: + msg371665
2020-05-15 05:01:25shihai1991setmessages: + msg368926
2020-05-14 21:48:08vstinnersetnosy: + p-ganssle
messages: + msg368869
2020-04-21 15:41:57vstinnersetmessages: + msg366924
2020-04-21 15:23:18shihai1991setmessages: + msg366921
2020-04-20 19:06:26brett.cannonsetmessages: + msg366867
2020-04-18 05:24:34shihai1991setnosy: + brett.cannon, ammar2
messages: + msg366693
2020-04-17 21:12:34vstinnersetmessages: + msg366679
2020-04-11 10:51:39shihai1991setmessages: + msg366199
2020-04-11 10:41:23shihai1991setmessages: + msg366198
2020-04-10 16:58:14shihai1991setmessages: + msg366141
2020-04-10 16:55:38shihai1991setmessages: + msg366140
2020-04-10 16:49:36shihai1991setkeywords: + patch
stage: patch review
pull_requests: + pull_request18816
2020-04-10 13:26:52corona10setnosy: + corona10
2020-04-09 15:18:54shihai1991setnosy: + shihai1991
2020-04-09 12:52:10vstinnercreate