classification
Title: "coverage" of Python regrtest cannot see initial import of libs
Type: Stage: patch review
Components: Demos and Tools Versions: Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brandon-rhodes, brett.cannon, eric.araujo, eric.snow, ezio.melotti, ncoghlan, nedbat, sandro.tosi, twouters
Priority: normal Keywords: patch

Created on 2011-03-15 22:09 by brandon-rhodes, last changed 2011-09-06 17:50 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
fullcoverage.patch brandon-rhodes, 2011-03-15 22:10 review
fullcoverage2.patch brandon-rhodes, 2011-08-01 19:30 Second version of a full-coverage tool for Python's own test suite review
stdlib_landmark.diff twouters, 2011-09-06 15:00 review
Messages (13)
msg131051 - (view) Author: Brandon Rhodes (brandon-rhodes) * Date: 2011-03-15 22:09
When running the Python regression tests in "coverage", the initial outer level of interpreted code in several standard library modules shows as not having been covered by the tests, because they were imported during the Python boot process and were already loaded when the "coverage" command got control.
msg131053 - (view) Author: Brandon Rhodes (brandon-rhodes) * Date: 2011-03-15 22:10
Here is a module that solves this problem if the tests are run with the "fullcoverage" directory at the front of the PYTHONPATH, like this:

PYTHONPATH=Tools/fullcoverage ./python -m coverage run --pylib Lib/test/regrtest.py test_copy
msg140135 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-11 15:50
Thanks for your work on this.  I made some minor comments on Rietveld (you should get an email), waiting for the real import experts to comment on the meat of the patch.
msg140164 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-07-11 21:57
Which I have not forgotten about. Just waiting until I have the time to get
to this.

On Mon, Jul 11, 2011 at 08:50, Éric Araujo <report@bugs.python.org> wrote:

>
> Éric Araujo <merwok@netwok.org> added the comment:
>
> Thanks for your work on this.  I made some minor comments on Rietveld (you
> should get an email), waiting for the real import experts to comment on the
> meat of the patch.
>
> ----------
> nosy: +eric.araujo
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11561>
> _______________________________________
>
msg141529 - (view) Author: Brandon Rhodes (brandon-rhodes) * Date: 2011-08-01 19:30
Éric, I think your suggestions are all good ones, and I have incorporated them into the file. (But do note that the departures we are now making from Ned's own copy of the tracer code ­— removing the commented-out debugging statement, and the long comment, and the inheritance from object — might make it harder to bring in changes from his own copy if he should ever further improve it.) I have tried to write the comments to be more informative, while also addressing your own ideas; let me know if you like the result!

Oh: and, I am continuing to use this new file in my own work on the Python core, and it has been working fine — so no problems with the actual code have developed over these first 3+ months of use.
msg141577 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-02 18:29
> I think your suggestions are all good ones, and I have incorporated
> them into the file.
Great :)  You left some commented-out debugging prints.

> (But do note that the departures we are now making from Ned's own
> copy of the tracer code ­— removing the commented-out debugging
> statement, and the long comment, and the inheritance from object —
> might make it harder to bring in changes from his own copy if he
> should ever further improve it.)
I’m not concerned by this.  I’ve been doing merges all the time for more that a year, and having a few diverging lines is not a big deal.

> I have tried to write the comments to be more informative, while also
> addressing your own ideas; let me know if you like the result!
I do!

> Oh: and, I am continuing to use this new file in my own work on the
> Python core, and it has been working fine — so no problems with the
> actual code have developed over these first 3+ months of use.
This is a good thing to know.

I’ll let Brett take over now.
msg142657 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-08-22 02:08
Added Ned to the nosy list.

For Python core, we should also keep in mind that we do have the option of adding a -X coverage option to avoid the need for the encodings module hack.
msg142659 - (view) Author: Ned Batchelder (nedbat) * Date: 2011-08-22 02:18
The tip of the coverage.py repo (https://bitbucket.org/ned/coveragepy) has a new implementation of the encodings hack which seems to work well.

Of course, an option to run a module before anything else in the interpreter would make the whole thing much simpler.
msg143582 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-09-06 04:48
Brandon, can I go ahead and close this and consider the script you and Ned have in the coverage.py repo to be the canonical script to use?
msg143584 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-06 05:32
Do we really want to enshrine shadowing the encodings module as the one true way to do this kind of thing?

Although I guess defining a way to do it properly would be a fairly major undertaking, so perhaps blessing the encodings hack is the practical way forward...

If we do that, I'd like to see a devguide patch that better explains how to generate your own coverage data for the stdlib with coverage.py, the encodings hack and the standard library coverage option.
msg143603 - (view) Author: Brandon Rhodes (brandon-rhodes) * Date: 2011-09-06 13:09
Brett, yes, you are welcome to close this issue — Ned quite handily convinced me that coverage code belongs in the "coverage" distribution, not languishing about in the CPython source tree. That solution also quite beautifully solves the copyright problem. So I happily withdraw my request for this feature.

Nick, Brett is working on exactly the sort of devguide improvement that you suggest — not least because the devguide will now need to instruct people in how to build "coverage" so that its C-accelerated tracer is available, which Ned's own patch to "coverage" to cover stdlib tracing uses instead of the Python tracer that I cut-and-pasted into this patch.

Finally, it would be wonderful to have a more formal mechanism for boot-time interventions. I have mentioned before my wish that Python's first action be to open() the executable, check its tail to see if it's a valid zipfile, and if so to try loading and running "startup.py" from that zipfile. Among other things, that would allow single-file distribution of pure-Python applications without the py2exe/py2app mess that prevails in the projects I work with today. But since the whole issue of grabbing control at boot time raises hackles ("why would you want to do that!?"), and I needed something working immediately during the PyCon sprint, I elected to simply adopt "encodings.py" as my way in. It works great, and "coverage" can evolve to an even better mechanism as soon as one becomes available, should anyone want to take the bootup option and run with it.

One final thought: should PyPy etc also implement the same boot protocol, should one be invented, so that all mainline interpreters can be instrumented the same way?
msg143610 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2011-09-06 15:00
For what it's worth, the need for a bootstrap-module has also come up within Google, where we have... somewhat different requirements than most.  In order to fix import paths in a way that works even when using "python -S", I had a need to patch the very first module that gets imported. Unfortunately the actual first module that gets imported depends on the arguments passed to Python. So, I wrote the attached stdlib_landmark.diff patch, which makes a 'stdlib_landmark.py' file that is both the stdlib landmark (used by Python to find the stdlib itself, currently 'os.py') *and* the very first module that is imported, always.

(The 'stdlib_landmark' name may be a little clunky; a better name, mirroring sitecustomize, may be "stdlibcustomize" or such.)

And yes, this could have been fixed in other ways. I could've patched the interpreter directly, or made Python understand symlinks to .py/.pyc files better, or tried to make a bunch of other tools work better with symlinks. This has turned out the most convenient solution for a number of reasons, though.
msg143635 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-09-06 17:50
Going to close this and open a separate feature request to better control what module is imported first.
History
Date User Action Args
2011-09-06 17:50:07brett.cannonsetstatus: open -> closed
resolution: out of date
messages: + msg143635
2011-09-06 15:00:13twouterssetfiles: + stdlib_landmark.diff
nosy: + twouters
messages: + msg143610

2011-09-06 13:09:53brandon-rhodessetmessages: + msg143603
2011-09-06 05:32:15ncoghlansetstatus: pending -> open

messages: + msg143584
2011-09-06 04:48:12brett.cannonsetstatus: open -> pending

messages: + msg143582
2011-09-06 04:47:02brett.cannonlinkissue12907 dependencies
2011-08-22 02:18:55nedbatsetmessages: + msg142659
2011-08-22 02:08:33ncoghlansetnosy: + nedbat
messages: + msg142657
2011-08-04 23:29:33ezio.melottisetnosy: + ezio.melotti
2011-08-02 18:29:58eric.araujosetmessages: + msg141577
2011-08-02 18:22:23eric.araujosetfiles: - unnamed
2011-08-01 19:31:00brandon-rhodessetfiles: + fullcoverage2.patch

messages: + msg141529
2011-07-11 21:57:35brett.cannonsetfiles: + unnamed

messages: + msg140164
2011-07-11 16:21:53eric.snowsetnosy: + eric.snow
2011-07-11 15:50:01eric.araujosetnosy: + eric.araujo
messages: + msg140135
2011-03-22 06:06:21sandro.tosisetnosy: + sandro.tosi
2011-03-22 03:08:01brett.cannonsetassignee: brett.cannon
versions: + Python 3.3
nosy: brett.cannon, ncoghlan, brandon-rhodes
components: + Demos and Tools
stage: patch review
2011-03-15 22:22:35ncoghlansetnosy: + brett.cannon, ncoghlan
2011-03-15 22:10:45brandon-rhodessetfiles: + fullcoverage.patch

messages: + msg131053
keywords: + patch
2011-03-15 22:09:10brandon-rhodescreate