This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_dis should test the dis module, not everything else
Type: enhancement Stage: patch review
Components: Tests Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: JelleZijlstra, Mark.Shannon, brandtbucher, iritkatriel, terry.reedy
Priority: normal Keywords: patch

Created on 2022-02-15 14:17 by Mark.Shannon, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 31369 merged Mark.Shannon, 2022-02-16 10:41
Messages (2)
msg413291 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2022-02-15 14:17
<rant>

This is getting really annoying.
It takes longer to fix all the heavily coupled and poorly written tests in test_dis than to make the real changes.

Tiny changes in the calling sequence, or reordering CFGs, cause huge diffs in the test_dis module.
No one ever checks these changes, they are just noise.

I've put this under "enhancement" as there is no "wastes a huge amount of time" category.

</rant>

The test_dis should not:

Contain offsets; they turn one line diffs into 100 line diffs
Contain tests for the compiler; they belong elsewhere.
Contain big strings; write proper tests not just regex matches.
Tests for Instruction should should not depend on the compiler output; create the bytecode directly.


This is not a new problem, but it does seem to be getting progressively worse.

A lot of the irritation stems from
https://github.com/python/cpython/commit/b39fd0c9b8dc6683924205265ff43cc597d1dfb9
although the tests from before that still hardcode offsets.
msg413514 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2022-02-18 21:38
With IDLE, I have issues with trying to test IDLE code without retesting tkinter, as well as deciding on the proper units of behavior to test.

Some suggestions: 
1. Add a docstring to the module with guidelines, after review from a couple of others.
For instance,I believe you are saying that no test should explicitly call compile.  Rather the test writer should call compile and extract the bytecode to copy into a test.

2. Make yourself a module code owner so you get informed and can review.  Reviewing would be easier with guidelines to refer to, instead of repeating them.

3. An example issue is that some callables take many types of arguments.  If not already, tests that the function can extract the bytecode from various objects should be separate from tests that extracted bytecode is then handled properly.  Would any internal refactoring of dis and addition of _test functions make it easier to make test more independent of each other?

4. I would rather read the multiple long lists like

expected_opinfo_outer = [
  Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=3, argrepr='3', offset=0, starts_line=2, is_jump_target=False),
  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, argrepr='4', offset=3, starts_line=None, is_jump_target=False),
  ...
]

condensed to

expected_opinfo_outer = [Instruction(opname, opcode, arg, argval, argrepr,
                                     offset, starts_line, is_jumps_target)
    for opname, opcode, arg, argval, argrepr, offset, starts_line, is_jumps_target in
(('LOAD_CONST', 100,    1,   3,     '3',      0,      2,           False),
 ('LOAD_CONST', 100,    2,   4,     '4',      3,      None,        False),
 ...
)]
History
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 90916
2022-02-18 21:38:39terry.reedysetnosy: + terry.reedy
messages: + msg413514
2022-02-16 17:40:11Mark.Shannonsetnosy: + brandtbucher, iritkatriel
2022-02-16 10:41:09Mark.Shannonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request29519
2022-02-15 17:50:58JelleZijlstrasetnosy: + JelleZijlstra
2022-02-15 14:17:08Mark.Shannoncreate