classification
Title: regrtest.py -T broken
Type: behavior Stage: patch review
Components: Tests Versions: Python 3.2
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: belopolsky, brett.cannon, doerwalter, haypo, pitrou
Priority: normal Keywords: patch

Created on 2010-11-26 16:02 by doerwalter, last changed 2013-01-12 14:27 by eli.bendersky.

Files
File name Uploaded Description Edit
build.log doerwalter, 2010-11-26 16:02 shell log
build2.log doerwalter, 2010-11-29 15:16 shell log 2
issue10541.diff belopolsky, 2010-12-14 23:03 review
Messages (9)
msg122463 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2010-11-26 16:02
Running regrtest.py with coverage option seems to be broken for the py3k branch at the moment. Run the following commands on the shell:

wget http://svn.python.org/snapshots/python3k.tar.bz2
tar xjf python3k.tar.bz2
cd python
./configure --enable-unicode=ucs4 --with-pydebug
make coverage
./python.exe Lib/test/regrtest.py -T -N test_urllib

This gives the following output:

[1/1] test_urllib
Not printing coverage data for 'Lib/test/regrtest.py': [Errno 2] No such file or directory: 'Lib/test/regrtest.py'
Traceback (most recent call last):
  File "Lib/test/regrtest.py", line 1502, in <module>
    main()
  File "Lib/test/regrtest.py", line 698, in main
    r.write_results(show_missing=True, summary=True, coverdir=coverdir)
  File "/Users/walter/x/pybug/python/Lib/trace.py", line 331, in write_results
    with open(filename, 'rb') as fp:
IOError: [Errno 2] No such file or directory: 'Lib/test/regrtest.py'
[123146 refs]

I'm testing on Mac OS X 10.6.5.

Attached is the complete log of the shell session.

This bug might be related to issue 10329, as the failing line was introduced in r86303.
msg122508 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-27 08:32
Apparently something in the test changes the cwd.

Both of the following invocations work:

$ ./python.exe `pwd`/Lib/test/regrtest.py -T -N test_urllib
$ ./python.exe -m test.regrtest -T -N test_urllib

I would suggest changing the coverage target in the Makefile.
msg122544 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-27 19:42
`make coverage` is fine, you just have to use the "-m test.regrtest" form when running the tests.
msg122545 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-27 19:45
I would like to investigate this some more.  In theory, regrtest should restore cwd before coverage results are written.
msg122810 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2010-11-29 15:16
OK, I reran the test with::

   ./python -mtest.regrtest -T -N test_urllib

and this does indeed produce coverage files (for _abcoll, _weakrefset, abc, base64, codecs, collections, contextlib, functools, genericpath, hashlib, locale, mimetypes, os, posixpath, quopri, random, re, sre_compile, sre_parse, ssl, stat, tempfile, textwrap, trace, uu, warnings).

However running the complete test suite via::

   ./python -mtest.regrtest -T -N -uurlfetch,largefile,network,decimal

fails with::

Not printing coverage data for '/tmp/tmp0fdr9o/t4/sub/subsub/__init__.py': [Errno 2] No such file or directory: '/tmp/tmp0fdr9o/t4/sub/subsub/__init__.py'
Traceback (most recent call last):
  File "/home/coverage/python/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/coverage/python/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/home/coverage/python/Lib/test/regrtest.py", line 1502, in <module>
    main()
  File "/home/coverage/python/Lib/test/regrtest.py", line 698, in main
    r.write_results(show_missing=True, summary=True, coverdir=coverdir)
  File "/home/coverage/python/Lib/trace.py", line 331, in write_results
    with open(filename, 'rb') as fp:
IOError: [Errno 2] No such file or directory: '/tmp/tmp0fdr9o/t4/sub/subsub/__init__.py'
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' encoding='ANSI_X3.4-1968'>

(attached is the complete output of running the test suite (build2.log).)
msg123952 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-14 15:19
Here is a simpler invocation that produces a similar error:

$ ./python.exe -m test.regrtest -T  test_trace test_pkg
...

IOError: [Errno 2] No such file or directory: '/var/folders/qs/qsqFUI2xFUKG+9CTf4z7pU+++TI/-Tmp-/tmpy1iyp7/t4/sub/__init__.py'
msg123990 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-14 23:03
I am attaching a patch that fixes write_results() and makes test_trace tests restore the tracefunc after they run.  This fixes generation off the coverage files, but many tests still fail when traced.
msg126864 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-01-22 23:45
At least the test_trace fix for preventing the complete destruction of any preset trace should be looked at and possibly applied separately (prevents coverage.py from covering the entire test suite as well). Should probably add a test to regrtest to make sure that the trace function is not mutated after a test file runs.
msg179701 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-01-11 16:31
Alexander's patch LGTM
History
Date User Action Args
2013-01-12 14:27:47eli.benderskysetnosy: - eli.bendersky
2013-01-11 16:31:13brett.cannonsetassignee: belopolsky
messages: + msg179701
2011-01-23 02:58:15terry.reedysetnosy: - terry.reedy
2011-01-22 23:45:55brett.cannonsetnosy: + brett.cannon
messages: + msg126864
2010-12-15 19:31:05belopolskysetnosy: + terry.reedy, eli.bendersky
2010-12-14 23:03:43belopolskysetfiles: + issue10541.diff
versions: + Python 3.2
messages: + msg123990

keywords: + patch
stage: needs patch -> patch review
2010-12-14 15:19:35belopolskysetmessages: + msg123952
2010-12-14 02:45:35r.david.murraysettype: behavior
stage: needs patch
2010-11-29 18:16:59pitrousetassignee: haypo -> (no value)
resolution: not a bug ->
2010-11-29 15:16:07doerwaltersetfiles: + build2.log

messages: + msg122810
2010-11-27 19:45:54belopolskysetstatus: pending -> open

messages: + msg122545
2010-11-27 19:42:22pitrousetstatus: open -> pending

nosy: + pitrou
messages: + msg122544

resolution: not a bug
2010-11-27 08:32:17belopolskysetmessages: + msg122508
2010-11-27 01:30:28hayposetnosy: + belopolsky
2010-11-26 16:02:12doerwaltercreate