classification
Title: Report correct coverage.py data for tests that invoke subprocesses
Type: enhancement Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, jaysinh.shukla, ncoghlan, nedbat
Priority: normal Keywords:

Created on 2011-05-02 13:46 by ncoghlan, last changed 2017-07-15 15:46 by nedbat.

Pull Requests
URL Status Linked Edit
PR 1865 closed brett.cannon, 2017-05-30 19:32
Messages (9)
msg134967 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-05-02 13:46
http://nedbatchelder.com/code/coverage/subprocess.html describes how to instruct a test suite that spawns subprocesses to correctly report coverage data for code covered only in those subprocesses.

regrtest currently doesn't do this, so modules that use subprocesses to run their tests may report inaccurate coverage numbers when using this tool (as recommended in the devguide). (Those numbers are irredeemably inaccurate when it comes to regrtest's own coverage assessment)

It may be better to build explicit invocation of coverage.py when regrtest is run under coverage.py into test.script_helper rather than trying to use the implicit mechanism though (as neither sitecustomize nor a .pth file are particularly good ideas when running Python's own test suite).
msg134968 - (view) Author: Ned Batchelder (nedbat) * Date: 2011-05-02 13:48
Let me know if I can help.
msg291218 - (view) Author: Jaysinh shukla (jaysinh.shukla) * Date: 2017-04-06 09:03
I found the regrtest wasn't displaying correct coverage for when the code is executed from call `Lib.test.support.script_helper.assert_python_ok`. I found `assert_python_ok` is using `subprocess` under the hood. It seems this problem is unobserved from a long time but it is having status `open`.

What should be the ideal situation for this? I request any core-developer to guide on this. I am interested in doing some more research on this issue. Thanks!
msg291219 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-04-06 09:26
As a starting point, I'd suggest looking at what can be achieved without making any changes to CPython or its test suite:

1. Set COVERAGE_PROCESS_START in the environment where the tests are being run

2. Inject a sitecustomize.py file into Lib (and add `Lib/sitecustomize.py` to `.gitignore`)

There are cases that won't cover (like subprocesses with a custom environment), but it will provide a starting point for the tests that just pass the current environment through, and will also provide a way to notify test.support.script_helper of the expected value of COVERAGE_PROCESS_START in the future.
msg291221 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-04-06 09:39
To be more specific regarding `sitecustomize.py`:

    $ echo "print('Hello from sitecustomize.py')" > Lib/sitecustomize.py
    $ ./python -c "print('Hello from command line')"
    Hello from sitecustomize.py
    Hello from command line

Since we only need these instructions to work for a local checkout, we can rely on the `sitecustomize.py` hook.

It means we'll still miss coverage results from subprocess tests run in isolated mode or with site.py processing disabled, but those are both relatively rare and involve *not* running code that is normally run, so shouldn't impact the aggregate coverage results.
msg291242 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-04-06 17:57
There's also what changes might occur in the coverage results when we start using fullcoverage: https://github.com/python/core-workflow/issues/18
msg294672 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-05-29 05:13
I learned a few thing trying to make this work.

One is that the COVERAGE_PROCESS_START does need to point to an actual .ini file (which includes an empty file).

Two, https://bitbucket.org/ned/coveragepy/src/63dfe482a849e9cb0b82214242315a806a6a3d53/coverage/control.py?at=default&fileviewer=file-view-default#control.py-1265 shows that stdlib coverage is left off by this, so even if you do turn on tracing in a subprocess, it will still ignore the stdlib. That means the coverage config file must turn on stdlib coverage in order or it to be picked up in the subprocesses.

Three, you need to run `coverage combine` to create a unified .coverage file for `coverage report` to pick up (I don't know if codecov explicitly needs the merge step.

When I have time I'll make a PR to test if this change actually helps speed things up.
msg296735 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-06-23 18:41
Just an FYI that I have never managed to make this work.
msg298399 - (view) Author: Ned Batchelder (nedbat) * Date: 2017-07-15 15:46
Whoever takes this up next: let me know if I can help.
History
Date User Action Args
2017-07-15 15:46:43nedbatsetmessages: + msg298399
2017-06-23 18:41:13brett.cannonsetassignee: brett.cannon ->
2017-06-23 18:41:06brett.cannonsetmessages: + msg296735
2017-05-30 19:32:30brett.cannonsetpull_requests: + pull_request1948
2017-05-29 05:13:32brett.cannonsetassignee: brett.cannon
2017-05-29 05:13:21brett.cannonsetmessages: + msg294672
2017-04-06 17:57:43brett.cannonsetmessages: + msg291242
2017-04-06 09:39:25ncoghlansetmessages: + msg291221
2017-04-06 09:26:01ncoghlansetmessages: + msg291219
2017-04-06 09:03:28jaysinh.shuklasetnosy: + jaysinh.shukla
messages: + msg291218
2011-05-02 13:48:52nedbatsetmessages: + msg134968
2011-05-02 13:46:09ncoghlancreate