This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Early tracing has lineno=None for modules
Type: Stage:
Components: Interpreter Core Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, nedbat
Priority: normal Keywords: 3.11regression

Created on 2022-03-03 13:36 by nedbat, last changed 2022-04-11 14:59 by admin.

Messages (3)
msg414438 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2022-03-03 13:36
Coverage.py has a trick to measure the early execution of stdlib modules. It has an "encodings.py" file that sets a trace function, then gets out of the way to let the real encodings.py execute.  In 3.11.0a5, that early trace function gets None values for the line numbers when modules were first executed.

To reproduce, create two files, encodings.py and prog.py:

--- 8< ------------------------------------------
# encodings.py
import sys

class FullCoverageTracer:
    def __init__(self):
        self.traces = []

    def fullcoverage_trace(self, *args):
        frame, event, arg = args
        if frame.f_lineno is None:
            self.traces.append((frame.f_code.co_name, frame.f_code.co_filename))
        return self.fullcoverage_trace

sys.settrace(FullCoverageTracer().fullcoverage_trace)

parentdir = max(filter(__file__.startswith, sys.path), key=len)
sys.path.remove(parentdir)
del sys.modules['encodings']
import encodings
-------------------------------------------------

--- 8< ------------------------------------------
# prog.py
import sys

print(sys.version)
trace = sys.gettrace()
print("None traces:", trace.__self__.traces)
print("Hello")
-------------------------------------------------

Then run:

% PYTHONPATH=$(pwd) python3.10 prog.py
3.10.2 (main, Jan 15 2022, 05:51:59) [Clang 12.0.0 (clang-1200.0.32.29)]
None traces: []
Hello

% PYTHONPATH=$(pwd) python3.11 prog.py
3.11.0a5 (main, Feb  3 2022, 19:11:58) [Clang 12.0.0 (clang-1200.0.32.29)]
None traces: [('<module>', '/usr/local/pyenv/pyenv/versions/3.11.0a5/lib/python3.11/encodings/__init__.py'), ('<module>', '<frozen codecs>'), ('<module>', '/usr/local/pyenv/pyenv/versions/3.11.0a5/lib/python3.11/encodings/aliases.py'), ('<module>', '/usr/local/pyenv/pyenv/versions/3.11.0a5/lib/python3.11/encodings/utf_8.py'), ('<module>', '<frozen io>'), ('<module>', '<frozen abc>'), ('<module>', '<frozen site>'), ('<module>', '<frozen os>'), ('<module>', '<frozen stat>'), ('<module>', '<frozen _collections_abc>'), ('<module>', '<frozen posixpath>'), ('<module>', '<frozen genericpath>'), ('<module>', '<frozen _sitebuiltins>'), ('<module>', '<string>'), ('<module>', '/System/Volumes/Data/root/src/bugs/fullcov/prog.py')]
Hello
msg414516 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2022-03-04 11:41
This is a bit of a tricky one.

The problem is that the line number for an instruction is used for two purposes.
1. To calculate the line number in frame.f_lineno for tracebacks and events
2. By dis to determine which lines are present and where they start.

If we set the lineno for the initial RESUME instruction in a module, we break dis and coverage for modules that do not have code on line 1.

If we add an explicit line-start table, we would be able to fix this, as the RESUME would have a line number for case 1, but wouldn't start a line, thus not breaking dis.

An explicit line-start table has other advantages, so we probably will implement it, but maybe not until 3.12.

Ned,
Would it make sense in coverage.py to ignore "call" events when doing coverage, and only use "line" events?
All the traces in your example are from "call" events.
msg414602 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2022-03-05 23:50
Maybe I'm missing something during normal execution, but I'm only seeing this effect now during this super-early encodings.py trick.  I don't mind just special-casing the Nones in this case.
History
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 91067
2022-03-05 23:50:12nedbatsetmessages: + msg414602
2022-03-04 11:41:51Mark.Shannonsetmessages: + msg414516
2022-03-03 13:36:59nedbatcreate