classification
Title: trace module cannot produce coverage reports for zipped modules
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: 10371 Superseder:
Assigned To: belopolsky Nosy List: Claudiu.Popa, belopolsky, terry.reedy, vstinner
Priority: normal Keywords: needs review, patch

Created on 2010-11-06 15:26 by belopolsky, last changed 2016-09-10 18:41 by belopolsky.

Files
File name Uploaded Description Edit
testtrace.zip belopolsky, 2010-11-06 15:26 run as "python testtrace.zip"
issue10342.diff belopolsky, 2010-11-08 01:32 review
issue10342a.diff belopolsky, 2014-06-30 01:01 review
Messages (12)
msg120626 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-06 15:26
Please run attached zip archive as a python script.  Note that the problem is not specific for using __main__.py - any module that comes from a zip archive or loaded by a custom loader would show the same bug. 

$ unzip -l test.zip 
Archive:  test.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
       79  10-24-08 18:26   __main__.py
 --------                   -------
       79                   1 file
$ $ cat __main__.py
from trace import Trace

def traced(flag):
    if flag:
        return 1
    else:
        return 2

tracer = Trace()
tracer.runfunc(traced, False)
results = tracer.results()
results.write_results(coverdir='.')

$ python testtrace.zip 
 --- modulename: __main__, funcname: traced
Not printing coverage data for 'testtrace.zip/__main__.py': [Errno 20] Not a directory: 'testtrace.zip/__main__.py'
__main__.py(4): __main__.py(7): $
msg120691 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-11-07 15:56
I commited Alexander's fix for #10329: r86303. The trace module now uses the input Python script encoding, instead of the locale encoding.
msg120701 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-08 01:32
I am attaching a "proof of concept" patch.  The trace code is in a dire need of restructuring to eliminate repeated reading of source files.
msg120703 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-11-08 01:47
+            try:
+                with open(filename, 'rb') as fp:
+                    encoding, _ = tokenize.detect_encoding(fp.readline)
+            except IOError:
+                encoding = None

You should use 'utf-8' instead of None (which will fall back to the locale encoding) here.
msg120704 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-08 01:57
On Sun, Nov 7, 2010 at 8:47 PM, STINNER Victor <report@bugs.python.org> wrote:
>
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
>
> +            try:
> +                with open(filename, 'rb') as fp:
> +                    encoding, _ = tokenize.detect_encoding(fp.readline)
> +            except IOError:
> +                encoding = None
>
> You should use 'utf-8' instead of None (which will fall back to the locale encoding) here.
>

I know.  I was too lazy to look up the correct spelling for the proof
of concept.  I am really posting this patch to show how this bug can
be fixed in theory and discuss how much of refactoring is acceptable.
For example, do we need to preserve trace.find_strings() function?
What is the best way to pass around source code? - file-like objects,
line iterators, readline-like function?  Also compile(prog, filename,
"exec") to find module's bytecode is a hack.  There must be a standard
way to achieve that which would use a .pyc file if available.
msg120707 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-11-08 01:59
The handling of __loader__ looks fine to me, although I don't quite see the point of the get_source call as it isn't returned or used.
msg120708 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-08 02:03
On Sun, Nov 7, 2010 at 8:59 PM, Brett Cannon <report@bugs.python.org> wrote:

> .. I don't quite see the point of the get_source call as it isn't returned or used.

It is passed to find_docstrings() to produce the "strs" dictionary.
msg121051 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-11-12 17:54
>What is the best way to pass around source code?
> - file-like objects, line iterators, readline-like function?

Line iterator (list of lines) as returned by open().readlines.
Memory should not be an issue. Read disk once and close.

with open('file', ...) as f:
  src = f.readlines()
msg121056 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-12 19:09
On Fri, Nov 12, 2010 at 12:54 PM, Terry J. Reedy <report@bugs.python.org> wrote:
..
>>What is the best way to pass around source code?
>> - file-like objects, line iterators, readline-like function?
>
> Line iterator (list of lines) as returned by open().readlines.
> Memory should not be an issue. Read disk once and close.
>
> with open('file', ...) as f:
>  src = f.readlines()

Line iterator does not carry the information about source encoding
which may be important for annotating the source code.
msg121067 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-11-12 20:07
>Line iterator does not carry the information about source encoding
which may be important for annotating the source code.

I would pass around both encoding and lines, possibly as a tuple.

A person heavily into OO might define a _Source class and turn the private functions into methods. But that would not play well with the deprecation process until it is done with.
msg221928 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-30 01:01
I updated the patch for 3.5.

[Brett]
> I don't quite see the point of the get_source call as it isn't returned or used.

It *is* used: it is passed to _find_strings_stream:

+        source = loader.get_source(modulename)
+        strs = _find_strings_stream(io.StringIO(source))
msg221944 - (view) Author: Claudiu Popa (Claudiu.Popa) * (Python triager) Date: 2014-06-30 06:55
Hi, I left a couple of comments on Rietveld.
History
Date User Action Args
2016-09-10 18:41:24belopolskysetversions: + Python 3.7, - Python 3.5
2014-06-30 06:55:17Claudiu.Popasetnosy: + Claudiu.Popa
messages: + msg221944
2014-06-30 01:01:11belopolskysetfiles: + issue10342a.diff

messages: + msg221928
versions: + Python 3.5, - Python 3.2
2013-02-01 22:42:06brett.cannonsetnosy: - brett.cannon
2012-07-15 03:54:18eli.benderskysetnosy: - eli.bendersky
2010-11-12 20:07:44terry.reedysetmessages: + msg121067
2010-11-12 19:09:05belopolskysetmessages: + msg121056
2010-11-12 17:54:02terry.reedysetmessages: + msg121051
2010-11-09 03:30:10belopolskysetdependencies: + Deprecate trace module undocumented API
2010-11-08 02:03:18belopolskysetmessages: + msg120708
2010-11-08 01:59:09brett.cannonsetmessages: + msg120707
2010-11-08 01:57:13belopolskysetmessages: + msg120704
2010-11-08 01:47:13vstinnersetmessages: + msg120703
2010-11-08 01:41:38belopolskysetnosy: + brett.cannon
2010-11-08 01:32:28belopolskysetfiles: + issue10342.diff

nosy: + terry.reedy, eli.bendersky
messages: + msg120701

keywords: + needs review, patch
stage: test needed -> patch review
2010-11-07 15:56:09vstinnersetmessages: + msg120691
2010-11-06 15:26:46belopolskycreate