classification
Title: Add figleaf coverage metrics
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.5
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, ajaksu2, eric.araujo, giampaolo.rodola, jerry.seutter, ncoghlan, nnorwitz, sandro.tosi
Priority: low Keywords: patch

Created on 2008-03-18 19:42 by jerry.seutter, last changed 2014-07-06 06:54 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
README.patch jerry.seutter, 2008-03-18 19:42 Update Tools/README
coverage.zip jerry.seutter, 2008-03-18 19:45 Tools/coverage directory containing files
coverage.zip jerry.seutter, 2008-03-18 22:39 Tools/coverage .zip revision 2
coverage.py ajaksu2, 2009-04-26 00:52 Main file in coverage.zip
Messages (14)
msg63975 - (view) Author: Jerry Seutter (jerry.seutter) * (Python committer) Date: 2008-03-18 19:42
This issue adds support for figleaf unit test coverage information.  The
diffs apply against trunk
msg63976 - (view) Author: Jerry Seutter (jerry.seutter) * (Python committer) Date: 2008-03-18 19:51
To test:

1. Unzip the zipfile in the base python directory.  The zipfile will
create Tools/coverage*.
2. cd Tools; patch -p0 README.patch
3. cd coverage
4. ../../python.exe coverage.py

The script will download figleaf, then run regrtest.py.  Any extra stuff
on the command line will be supplied to regrtest.
msg63979 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2008-03-18 20:19
Thanks for the patch.  It would be nice to get more instrumentation like
coverage, performance, etc.

Here are some things I noticed while reviewing the patch:

 * This won't work on unix other than OSX.  Can you change
../../python.exe to sys.executable?  
 * How big is figleaf?  Should you try to read/write the file in chunks?
 * optparse doesn't seem to be used.
 * Can you change `cat file` to read the file and pass those as
arguments so this could work on windows?

BTW, does figleaf work on Windows?
msg64005 - (view) Author: Jerry Seutter (jerry.seutter) * (Python committer) Date: 2008-03-18 22:39
Thanks for the input.

 * ../../python.exe changed to sys.executable.
 * Figleaf is 69kb.  It seems to work fine doing it all in one read()
call.  Should it be chunked?
 * Optparse isn't used, partially because I'm lazy and partly because
I'm not actually parsing the command line, just chopping the first
element off and then passing it off to regrtest.py.  It is the job of
regrtest.py to parse the command line. <-- My opinion
 * traceless.  I think I should remove the file.  Having tests that fail
don't hurt figleaf, as far as I know, so there isn't much point to
avoiding failing tests.

Uploading a new version of the .zip file that uses sys.executable and
does not use traceless.

I don't know if figleaf works on windows and I don't have a system to
test with.
msg64008 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2008-03-18 22:43
>   * ../../python.exe changed to sys.executable.
>   * Figleaf is 69kb.  It seems to work fine doing it all in one read()
>  call.  Should it be chunked?

At 69kb, nah.  It should be good enough for the first cut.

>   * Optparse isn't used, partially because I'm lazy and partly because
>  I'm not actually parsing the command line, just chopping the first
>  element off and then passing it off to regrtest.py.  It is the job of
>  regrtest.py to parse the command line. <-- My opinion

Oh sorry, I only meant to remove the import since it wasn't used.  I
wasn't secretly trying to get you to do more work (at least not yet).
:-)
msg64025 - (view) Author: Skip Montanaro (skip.montanaro) * Date: 2008-03-19 02:16
How will this work if I use a build directory?  For example,
my source is in ~/src/python/trunk.  In there I create a
build directory.  When I run it I get

% pwd
/Users/skip/src/python/trunk/build
% ./python.exe ../Tools/coverage/coverage.py 
Running tests...
Traceback (most recent call last):
  File "figleaf-latest/bin/figleaf", line 4, in <module>
    figleaf.main()
  File
"/Users/skip/src/python/trunk/build/figleaf-latest/figleaf/__init__.py",
line 302, in main
    execfile(sys.argv[0], __main__.__dict__)
IOError: [Errno 2] No such file or directory: '../../Lib/test/regrtest.py'
Generating html...
CANNOT OPEN: @test
figleaf: HTML output written to ../../coverage

I think this mode should work.  I prefer not to pollute my
source tree with build information.

Skip
msg64053 - (view) Author: Skip Montanaro (skip.montanaro) * Date: 2008-03-19 11:26
I gave this a try.  It seems to not report on many files.  For example,
test_csv was run and passed, but there is no html file in the coverage
directory with "csv" in its name after figleaf2html is run.  Nor is there a
key in the pickled dictionary in the .figleaf file which includes the string
"csv".

Skip
msg86580 - (view) Author: Daniel Diniz (ajaksu2) Date: 2009-04-26 00:52
Re-attaching the coverage bootstrapping file as plain/text.
msg112228 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-07-31 23:43
IIUC, using runpy (either from Python or in a subprocess i.e. python -m test.regrtest) would allow running from build dir.

Coverage reports are nice, but I personally dislike the idea of code downloading something via HTTP and running it. I’d prefer just having an optional dependency on it and letting the user install it if they want.

Code review:
1) There is already a popular (I think) module named coverage, please use another name.
1.1) How does figleaf compare with coverage?
2) Please use tarfile instead of running tar (not always available).
3) Please use the Python API instead of running subprocesses.
4) subprocess is recommended over os.system. Neither will be required if you agree to 2 and 3 :)
5) Please make the code PEP 8-compliant.

Hope this helps! :)
msg112229 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-07-31 23:45
the Python API of figleaf*
msg154941 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-03-05 10:54
Adding people interested in code coverage to nosy.
msg154987 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2012-03-05 22:49
The devguide reports a long section about code convering cpython: http://docs.python.org/devguide/coverage.html . It uses Ned's coverege.py and has a lot of details about how to use it and how to generate nice reports out of it. do we need an internal tool for cpython coverage?
msg213358 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-13 07:00
Indeed, the devguide documents how to use coverage.py, and there is even a pure-stdlib solution with regrtest.  I recommend closing this.
msg222383 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-06 00:00
As there have been no replies to msg213358 can someone close this please.
History
Date User Action Args
2014-07-06 06:54:21berker.peksagsetstatus: open -> closed
2014-07-06 00:00:47BreamoreBoysetstatus: pending -> open
versions: + Python 3.5, - Python 3.3
nosy: + BreamoreBoy

messages: + msg222383
2014-03-13 07:00:24eric.araujosetstatus: open -> pending
resolution: rejected
messages: + msg213358

stage: test needed -> resolved
2012-03-05 22:49:34sandro.tosisetnosy: + sandro.tosi
messages: + msg154987
2012-03-05 10:54:29eric.araujosetnosy: + ncoghlan, giampaolo.rodola

messages: + msg154941
versions: + Python 3.3, - Python 3.1, Python 2.7
2010-07-31 23:45:12eric.araujosetkeywords: patch, patch

messages: + msg112229
2010-07-31 23:43:55eric.araujosetkeywords: patch, patch
nosy: + eric.araujo
messages: + msg112228

2010-05-20 20:29:23skip.montanarosetkeywords: patch, patch
nosy: - skip.montanaro
2009-04-26 00:52:39ajaksu2setfiles: + coverage.py

type: enhancement
versions: + Python 3.1, Python 2.7, - Python 2.6
keywords: patch, patch
nosy: + ajaksu2

messages: + msg86580
stage: test needed
2008-03-19 11:26:44skip.montanarosetmessages: + msg64053
2008-03-19 02:16:23skip.montanarosetkeywords: patch, patch
nosy: + skip.montanaro
messages: + msg64025
2008-03-18 22:43:41nnorwitzsetmessages: + msg64008
2008-03-18 22:39:52jerry.seuttersetkeywords: patch, patch
files: + coverage.zip
messages: + msg64005
2008-03-18 20:19:41nnorwitzsetkeywords: patch, patch
nosy: + nnorwitz
messages: + msg63979
2008-03-18 19:51:37jerry.seuttersetkeywords: patch, patch
messages: + msg63976
2008-03-18 19:45:37jerry.seuttersetkeywords: patch, patch
files: + coverage.zip
2008-03-18 19:42:54jerry.seuttercreate