This issue adds support for figleaf unit test coverage information.  The
diffs apply against trunk

To test:

1. Unzip the zipfile in the base python directory.  The zipfile will
create Tools/coverage*.
2. cd Tools; patch -p0 README.patch
3. cd coverage
4. ../../python.exe coverage.py

The script will download figleaf, then run regrtest.py.  Any extra stuff
on the command line will be supplied to regrtest.

Thanks for the patch.  It would be nice to get more instrumentation like
coverage, performance, etc.

Here are some things I noticed while reviewing the patch:

 * This won't work on unix other than OSX.  Can you change
../../python.exe to sys.executable?  
 * How big is figleaf?  Should you try to read/write the file in chunks?
 * optparse doesn't seem to be used.
 * Can you change `cat file` to read the file and pass those as
arguments so this could work on windows?

BTW, does figleaf work on Windows?

Thanks for the input.

 * ../../python.exe changed to sys.executable.
 * Figleaf is 69kb.  It seems to work fine doing it all in one read()
call.  Should it be chunked?
 * Optparse isn't used, partially because I'm lazy and partly because
I'm not actually parsing the command line, just chopping the first
element off and then passing it off to regrtest.py.  It is the job of
regrtest.py to parse the command line. <-- My opinion
 * traceless.  I think I should remove the file.  Having tests that fail
don't hurt figleaf, as far as I know, so there isn't much point to
avoiding failing tests.

Uploading a new version of the .zip file that uses sys.executable and
does not use traceless.

I don't know if figleaf works on windows and I don't have a system to
test with.

>   * ../../python.exe changed to sys.executable.
>   * Figleaf is 69kb.  It seems to work fine doing it all in one read()
>  call.  Should it be chunked?

At 69kb, nah.  It should be good enough for the first cut.

>   * Optparse isn't used, partially because I'm lazy and partly because
>  I'm not actually parsing the command line, just chopping the first
>  element off and then passing it off to regrtest.py.  It is the job of
>  regrtest.py to parse the command line. <-- My opinion

Oh sorry, I only meant to remove the import since it wasn't used.  I
wasn't secretly trying to get you to do more work (at least not yet).
:-)

How will this work if I use a build directory?  For example,
my source is in ~/src/python/trunk.  In there I create a
build directory.  When I run it I get

% pwd
/Users/skip/src/python/trunk/build
% ./python.exe ../Tools/coverage/coverage.py 
Running tests...
Traceback (most recent call last):
  File "figleaf-latest/bin/figleaf", line 4, in <module>
    figleaf.main()
  File
"/Users/skip/src/python/trunk/build/figleaf-latest/figleaf/__init__.py",
line 302, in main
    execfile(sys.argv[0], __main__.__dict__)
IOError: [Errno 2] No such file or directory: '../../Lib/test/regrtest.py'
Generating html...
CANNOT OPEN: @test
figleaf: HTML output written to ../../coverage

I think this mode should work.  I prefer not to pollute my
source tree with build information.

Skip

I gave this a try.  It seems to not report on many files.  For example,
test_csv was run and passed, but there is no html file in the coverage
directory with "csv" in its name after figleaf2html is run.  Nor is there a
key in the pickled dictionary in the .figleaf file which includes the string
"csv".

Skip

Re-attaching the coverage bootstrapping file as plain/text.

IIUC, using runpy (either from Python or in a subprocess i.e. python -m test.regrtest) would allow running from build dir.

Coverage reports are nice, but I personally dislike the idea of code downloading something via HTTP and running it. I’d prefer just having an optional dependency on it and letting the user install it if they want.

Code review:
1) There is already a popular (I think) module named coverage, please use another name.
1.1) How does figleaf compare with coverage?
2) Please use tarfile instead of running tar (not always available).
3) Please use the Python API instead of running subprocesses.
4) subprocess is recommended over os.system. Neither will be required if you agree to 2 and 3 :)
5) Please make the code PEP 8-compliant.

Hope this helps! :)

the Python API of figleaf*